Preprocessor method and apparatus

ABSTRACT

The invention generally relates to multimedia data processing, and more particularly, to processing operations performed prior to or in conjunction with data compression processing. A method of processing multimedia data includes receiving interlaced video frames, obtaining metadata for the interlaced video frames, converting the interlaced video frames to progressive video using at least a portion of the metadata; and providing the progressive video and at least a portion of the metadata to an encoder for use in encoding the progressive video. The method can also include generating spatial information and bi-directional motion information, for the interlaced video frames, and generating progressive video based on the interlaced video frames using the spatial and bi-directional motion information.

CLAIM OF PRIORITY UNDER 35 U.S.C. §119

The present Application for Patent claims priority to ProvisionalApplication No. 60/789,266 entitled “PREPROCESSOR FOR MULTIMEDIA DATA”filed Apr. 4, 2006, all of which are assigned to the assignee hereof andhereby expressly incorporated by reference, herein.

CLAIM OF PRIORITY UNDER 35 U.S.C. §120

The present Application for Patent is a continuation in part of patentapplication Ser. No. 11/528,141 entitled “CONTENT DRIVEN TRANSCODER THATORCHESTRATES MULTIMEDIA TRANSCODING USING CONTENT INFORMATION” filedSep. 26, 2006, now U.S. Pat. No. 8,879,856, and assigned to the assigneehereof and hereby expressly incorporated by reference herein.

REFERENCE TO CO-PENDING APPLICATIONS FOR PATENT

The present Application for Patent is related to U.S. patent applicationSer. No. 11/373,577 entitled “CONTENT CLASSIFICATION FOR MULTIMEDIAPROCESSING” filed on Mar. 10, 2006, assigned to the assignee hereof andhereby expressly incorporated by reference herein.

BACKGROUND

1. Field

The invention generally relates to multimedia data processing, and moreparticularly, to processing operations performed prior to or inconjunction with data compression processing.

2. Background

SUMMARY

Each of the inventive apparatuses and methods described herein hasseveral aspects, no single one of which is solely responsible for itsdesirable attributes. Without limiting the scope of this invention, itsmore prominent features will now be discussed briefly. After consideringthis discussion, and particularly after reading the section entitled“Detailed Description” one will understand how the features of thisinvention provides improvements for multimedia data processingapparatuses and methods.

In one aspect, a method of processing multimedia data comprisesreceiving interlaced video frames, converting the interlaced videoframes to progressive video, generating metadata associated with theprogressive video, and providing the progressive video and at least aportion of the metadata to an encoder for use in encoding theprogressive video. The method can further include encoding theprogressive video using the metadata. In some aspects, the interlacedvideo frames comprise NTSC video. Converting the video frames caninclude deinterlacing the interlaced video frames. In some aspects, themetadata can include bandwidth information, bi-directional motioninformation, a bandwidth ratio, a complexity value such as a temporal ora spatial complexity value or both, luminance information, and thespatial information can include luminance and/or chrominanceinformation. The method can also include generating spatial informationand bi-directional motion information for the interlaced video framesand generating the progressive video based on the interlaced videoframes using the spatial and bi-directional motion information. In someaspects converting the interlaced video frames comprises inversetelecining 3/2 pulldown video frames, and/or resizing the progressivevideo. The method can further comprise partitioning the progressivevideo to determine group of picture information, where the partitioningcan include shot detection of the progressive video. In some aspects,the method also includes progressive video with a denoising filter.

In another aspect, an apparatus for processing multimedia data caninclude a receiver configured to receive interlaced video frames, adeinterlacer configured to convert the interlaced video frames toprogressive video, and a partitioner configured to generate metadataassociated with the progressive video and provide the progressive videoand the metadata to an encoder for use in encoding the progressivevideo. In some aspects, the apparatus can further include an encoderconfigured to receive the progressive video from the communicationsmodule and encode the progressive video using the provided metadata. Thedeinterlacer can be configured to perform spatio-temporal deinterlacingand/or inverse telecining. The partitioner can be configured to performshot detection and generate compression information based on the shotdetection. In some aspects the partitioner can be configured to generatebandwidth information. The apparatus can also include a resamplerconfigured to resize a progressive frame. The metadata can includebandwidth information, bi-directional motion information, a bandwidthratio, luminance information, a spatial complexity value related tocontent, and/or a temporal complexity value related to content. In someaspects, the deinterlacer is configured to generate spatial informationand bi-directional motion information for the interlaced video framesand progressive video based on the interlaced video frames using spatialand bi-directional motion information.

Another aspect comprises an apparatus for processing multimedia dataincludes means for receiving interlaced video frames, means forconverting the interlaced video frames to progressive video, means forgenerating metadata associated with the progressive video, and means forproviding the progressive video and at least a portion of the metadatato an encoder for use in encoding the progressive video. In some aspectsthe converting means comprises an inverse teleciner and/or aspatio-temporal deinterlacer. In some aspects, the generating means isconfigured to perform shot detection and generate compressioninformation based on the shot detection. In some aspects the generatingmeans is configured to generate bandwidth information. In some aspects,the generating includes means for resampling to resize a progressiveframe.

Another aspect comprises a machine readable medium comprisinginstructions for processing multimedia data that upon execution cause amachine to receive interlaced video frames, convert the interlaced videoframes to progressive video, generate metadata associated with theprogressive video, and provide the progressive video and at least aportion of the metadata to an encoder for use in encoding theprogressive video.

Another aspect includes a processor comprising a configuration toreceive interlaced video, convert the interlaced, video to progressivevideo, generate metadata associated with the progressive video, andprovide the progressive video and at least a portion of the metadata toan encoder for use in encoding the progressive video. The conversion ofthe interlaced video can include a performing spatio-temporal deinterlacing. In some aspects, the conversion of the interlaced videocomprises performing inverse telecine. In some aspects, generation ofmetadata includes generating compression information based on detectingshot changes. In some aspects, generation of metadata includesdetermining compression information of the progressive video. In someaspects, the configuration includes a configuration to resample video togenerate a resized a progressive frame. In some aspects, the metadatacan include bandwidth information, bi-directional motion information,complexity information such as temporal or spatial complexityinformation based on content, and/or compression information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a communications system for deliveringstreaming multimedia data;

FIG. 2 is a block diagram of a digital transmission facility thatincludes a preprocessor;

FIG. 3A is a block diagram of an illustrative aspect of a preprocessor;

FIG. 3B is a flow diagram that illustrates a process for processingmultimedia data;

FIG. 3C is a block diagram illustrating means for processing multimediadata;

FIG. 4 is a block diagram illustrating operations of an exemplarypreprocessor;

FIG. 5 is a diagram of phase decisions in an inverse telecine process;

FIG. 6 is a flow diagram illustrating a process of inverting telecinedvideo;

FIG. 7 is an illustration of a trellis showing phase transitions;

FIG. 8 is a guide to identify the respective frames that are used tocreate a plurality of matrices;

FIG. 9 is a flow diagram illustrating how the metrics of FIG. 8 arecreated;

FIG. 10 is a flow diagram which shows the processing of the metrics toarrive at air estimated phase;

FIG. 11 is a dataflow diagram illustrating a system for generatingdecision variables;

FIG. 12 is a block diagram depicting variables that are used to evaluatethe branch information;

FIGS. 13A, 13B and 13C are flow diagrams showing how lower envelopes arecomputed;

FIG. 14 is a flow diagram showing the operation of a consistencydetector;

FIG. 15 is a flow diagram showing a process of computing an offset to adecision variable that is used to compensate for inconsistency in phasedecisions;

FIG. 16 presents the operation of inverse telecine after the pull downphase has been estimated.

FIG. 17 is a block diagram of a deinterlacer device;

FIG. 18 is a block diagram of another deinterlacer device;

FIG. 19 is drawing of a subsampling pattern of an interlaced picture;

FIG. 20 is a block diagram of a deinterlacer device that uses Wmedfiltering motion estimation to generate a deinterlaced frame;

FIG. 21 illustrates one aspect of an aperture for determining, staticareas of multimedia data;

FIG. 22 is a diagram illustrating one aspect of an aperture fordetermining slow-motion areas of multimedia data;

FIG. 23 is a diagram illustrating an aspect of motion estimation;

FIG. 24 illustrates two motion vector maps used in determining motioncompensation;

FIG. 25 is a flow diagram illustrating a method of deinterlacingmultimedia data;

FIG. 26 is a flow diagram illustrating a method of generating adeinterlaced frame using spatio-temporal information;

FIG. 27 is a flow diagram illustrating a method of performing motioncompensation for deinterlacing;

FIG. 28 is a block diagram of a preprocessor comprising a processorconfigured for shot detection and other preprocessing operationsaccording to some aspects;

FIG. 29 illustrates the relationship between encoding complexity C andallocated bits B;

FIG. 30 is a flow diagram that illustrates a process that operates on agroup of pictures and can be used in some aspects to encode video basedon shot detection in video frames;

FIG. 31 is a flow diagram illustrating a process for shot detection;

FIG. 32 is a flow diagram illustrating a process for determiningdifferent classifications of shots in video;

FIG. 33 is a flow diagram illustrating a process for assigning framecompression schemes to video frames based on shot detection results;

FIG. 34 is a flow diagram illustrating a process for determining abruptscene changes;

FIG. 35 is a flow diagram illustrating a process for determiningslowly-changing scenes;

FIG. 36 is a flow diagram illustrating a process for determining scenescontaining camera flashes;

FIG. 37 illustrates motion compensation vectors between a current frameand a previous frame MV_(P) and a current frame and a next frame MV_(N);

FIG. 38 is a graph illustrating a relationship for a variable used indetermining a frame difference metric;

FIG. 39 is a block diagram illustrating encoding data and calculatingresiduals;

FIG. 40 is a block diagram illustrating determining a frame differencemetric;

FIG. 41 is a flow diagram illustrating the procedure where compressiontypes are assigned to frames;

FIG. 42 illustrates an example of 1-D poly phase resampling;

FIG. 43 is a graphic illustrating a safe action area and a safe titlearea of a frame of data; and

FIG. 44 is a graphic illustrating a safe action area of a frame of data.

DETAILED DESCRIPTION

The following description includes details to provide a thoroughunderstanding of the examples. However, it is understood by one ofordinary skill in the art that the examples may be practiced even ifevery detail of a process or device in an example or aspect is notdescribed or illustrated herein. For example, electrical components maybe shown in block diagrams that do not illustrate every electricalconnection or every electrical element of the component in order not toobscure the examples in unnecessary detail. In other instances, suchcomponents, other structures and techniques may be shown in detail tofarther explain the examples.

Described herein are certain inventive aspects and aspects forpreprocessors and preprocessor operations methods that improve theperformance of existing preprocessing and encoding systems. Suchpreprocessors can process metadata and video in preparation forencodings including performing deinterlacing, inverse telecining,filtering, identifying shot types, processing and generating metadata,and generating bandwidth information. References herein to “one aspect,”“an aspect,” some aspects,” or “certain aspects” mean that one or moreof a particular feature, structure, or characteristic described inconnection with the aspect can be included in at least one aspect of apreprocessor system. The appearances of such phrases in various placesin the specification are not necessarily all referring to the sameaspect, nor are separate or alternative aspects mutually exclusive ofother aspects. Moreover, various features are described which may beexhibited by some aspects and not by others. Similarly, various stepsare described which may be steps for some aspects but not other aspects.

“Multimedia data” or “multimedia” as used herein is a broad term thatincludes video data (which can include audio data), audio data, or bothvideo data and audio data. “Video data” or “video” as used herein as abroad term, which refers to an image or one or more series or sequencesof images containing text, image, and/or audio data, and can be used torefer to multimedia data or the terms may be used interchangeably,unless otherwise specified.

FIG. 1 is a block diagram of a communications system 100 for deliveringstreaming multimedia. Such system finds application in the transmissionof digital compressed video to a multiplicity of terminals as shown inFIG. 1. A digital video source can be, for example, a digital cable orsatellite feed or an analog source that is digitized. The video sourceis processed in a transmission facility 120 where it is encoded andmodulated onto a earner for transmission through a network 140 to one ormore terminals 160. The terminals 160 decode the received video andtypically display at least a portion the video. The network 140 refersto any type of communication network, wired or wireless, suitable forthe transmission of encoded data. For example, the network 140 can be acell phone network, wired or wireless local area network (LAN) or a widearea network (WAN), or the Internet. The terminals 160 can be any typeof communication device capable of receiving and displaying data,including, but not limited to, cell phones, PDA's, in-home or commercialvideo display equipment, computers (portable, laptop, handheld, PC's,and larger server-based computer systems), and personal entertainmentdevices capable of using multimedia data.

FIGS. 2 and 3 illustrate sample aspects of a preprocessor 202. In FIG.2, preprocessor 202 is in a digital transmission facility 120. A decoder201 decodes encoded data from a digital video source and providesmetadata 204 and video 205 to the preprocessor 202. The preprocessor 202is configured to perform certain types of processing on the video 205and the metadata 204 and provide processed metadata 206 (e.g., baselayer reference frames, enhancement layer reference frames, bandwidthinformation, content information) and video 207 to an encoder 203. Suchpreprocessing of multimedia data can improve the visual clarity,anti-aliasing, and compression efficiency of the data. Generally, thepreprocessor 202 receives video sequences provided by the decoder 201and converts the video sequences into progressive video sequences forfurther processing (e.g., encoding) by aft encoder. In some aspects, thepreprocessor 202 can be configured for numerous operations, includinginverse telecine, deinterlacing, filtering (e.g., artifact removal,de-ringing, de-blocking, and de-noising), resizing (e.g., spatialresolution down-sampling from standard definition to Quarter VideoGraphics Array (QVGA)), and GOP structure generation (e.g., calculatingcomplexity map generation, scene change detection, and fade/flashdetection).

FIG. 3A illustrates a preprocessor 202 that is configured with modulesor components (collectively referred to here as “modules”) to performits preprocessing operations on received metadata 204 and video 205, andthen provide processed metadata 206 and progressive video 207 forfurther processing (e.g., loan encoder). The modules can be implementedin hardware, software, firmware, or a combination thereof. Thepreprocessor 202 can include various modules, including one or more orthe modules illustrated, which include inverse telecine 301,deinterlacer 302, denoiser 303, alias suppressor 304, resampler 303,deblocker/derringer 306, and a GOP partitioner 307, all describedfurther below. The preprocessor 202 can also include other appropriatemodules that may be used to process the video and metadata, includingmemory 308 and a communications module 309. A software module may residein RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, a hard disk, a removable disk, a CD-ROM, or any other form ofstorage medium known in the art. An exemplary storage medium is coupledto the processor, such that the processor can read information from, andwrite information to, the storage medium. In die alternative, thestorage medium may be integral to the processor. The processor and thestorage medium may reside in an ASIC. The ASIC may reside in a userterminal. In the alternative, the processor and the storage medium mayreside as discrete components in a user terminal.

FIG. 3B is a flow diagram that illustrates a process 300 for processingof multimedia data. Process 300 starts and proceeds to block 320 whereinterlaced video is received. Preprocessor 202 illustrated in FIG. 2 andFIG. 3 can perform this step. In some aspects, a decoder (e.g., decoder201 FIG. 2) can receive the interlaced data and the provide it topreprocessor 202. In some aspects, a data receiving module 330 shown inFIG. 3C which is a portion of a preprocessor 202 can perform this step.Process 300 then proceeds to block 322 where interlaced video isconverted to progressive video. Preprocessor 202 in FIG. 2 and FIG. 3A,and module 332 of FIG. 3C can perform this step. If the interlaced videohas been telecined, block 322 processing can include performing inversetelecining to generate progressive video. Process 300 then proceeds toblock 324 to generate metadata associated with the progressive video.The GOP Partitioner 307 in FIG. 3A and a module 334 in FIG. 3C canperform such processing. Process 300 then proceeds to block 326 wherethe progressive video and at least a portion of the metadata areprovided to an encoder for encoding (e.g., compression). Preprocessor202 shows in FIG. 2 and FIG. 3A, and module 336 in FIG. 3C can performthis step. After providing the progressive video and associated metadatato another component for encoding, process 300 can end.

FIG. 3C is a block diagram illustrating means for processing multimediadata. Shown here such means are incorporated in a preprocessor 202. Thepreprocessor 202 includes means for receiving video such as module 330.The preprocessor 202 also includes means for converting interlaced datato progressive video such as module 332. Such means can include, forexample, a spatial-temporal deinterlacer and/or an inverse teleciner.The preprocessor 202 also includes means for generating metadataassociated with the progressive video such as module 334. Such means caninclude the GOP partitioner 307 (FIG. 3A) which can generate varioustypes of metadata as described herein. The preprocessor 202 can alsoinclude means for providing the progressive video and metadata to anencoder for encoding as illustrated by module 336. Such means caninclude a communications module 309 illustrated in FIG. 3A in someaspects. As will be appreciated by one skilled in the art, such meanscan be implemented in many standard ways.

The preprocessor 202 can use obtained metadata (e.g., obtained from thedecoder 201 or from another source) for one or more of the preprocessingoperations. Metadata can include information relating to, describing, orclassifying the content of the multimedia data (“consent information”).In particular the metadata can include a content classification. In someaspects, the metadata does not include content information desired forencoding operations. In such cases, the preprocessor 202 can beconfigured to determine content information and use the contentinformation for preprocessing operations and/or provides the contentinformation to other components, e.g., the decoder 203. In some aspects,the preprocessor 202 can use such content information to influence GOPpartitioning, determine appropriate type of filtering, and/or determineencoding parameters that are communicated to an encoder.

FIG. 4 shows an illustrative example of process blocks that can beincluded in the preprocessor, and illustrates processing that can beperformed by the preprocessor 202. In this example, the preprocessor 202receives metadata and video 204, 205 and provides output data 206, 207comprising (processed) metadata and video to the encoder 228. Typically,there are three types of video that is received by the preprocessor.First, the received video can be progressive video and deinterlacingdoes not have to be performed. Second, the video data can be telecinedvideo, interlaced video converted from 24 fps movie sequences, in whichcase the video. Third, the video can be non-telecined interlaced video.Preprocessor 226 can process these types of video as described below.

At block 401, the preprocessor 202 determines if the received video 204,205 is progressive video. In some cases, this can be determined from themetadata if the metadata contains such information, or by processing ofthe video itself. For example, an inverse telecine process, describedbelow, can determine if the received video 205 is progressive video. Ifit is, the process proceeds to block 407 where filtering operations areperformed on the video to reduce noise, such as white Gaussian noise. Ifthe video is not progressive video, at block 401 the process proceeds toblock 404 to a phase detector.

Phase detector 604 distinguishes between video that originated in atelecine and that which began in a standard broadcast format. If thedecision is made that the video was telecined (the YES decision pathexiting phase detector 404), the telecined video is returned to itsoriginal format in inverse telecine 406. Redundant fields are identifiedand eliminated and fields derived from the same video frame are rewoveninto a complete image. Since the sequence of reconstructed film imageswere photographically recorded at regular intervals of 1/24 of a second,the motion estimation process performed in a GOP petitioner 412 or adecoder is more accurate using the inverse telecined images rather thanthe telecined data, which has an irregular time base.

In one aspect, the phase detector 404 makes certain decisions afterreceipt of a video frame. These decisions include: (i) whether thepresent video from a telecine output and the 3:2 pull down phase is oneof the five phases P₀, P₁, P₂, P₃, and P₄ shown in FIG. 5; and (ii) thevideo was generated as conventional NTSC. That decision is denoted asphase P₅. These decisions appear as outputs of phase detector 404 shownin FIG. 4. The path from phase detector 404 labeled “YES” actuates theinverse telecine 406, indicating that it has been provided with thecorrect pull down phase so that it can sort out the fields that wereformed from the same photographic image and combine them. The path fromphase detector 404 labeled “NO” similarly actuates the deinterlacer 405to separate an apparent NTSC frame into fields for optimal processing.Inverse telecine is further described in co-pending U.S. patentapplication Ser. No. 11/550,752 entitled “Inverse Telecine AlgorithmBased on State Machine” which is owned by the assignee hereof andincorporated by reference herein in its entirety.

The phase detector 404 can continuously analyze video frames becausedifferent types of video may be received at any time. As an exemplary,video conforming to the NTSC Standard may be inserted into the video asa commercial. After inverse telecine, the resulting progressive video issent to a denoiser (filter) 407 which can be used to reduce whiteGaussian noise.

When conventional NTSC video is recognized (the NO path from phasedetector 401), it is transmitted to deinterlacer 405 for compression.The deinterlacer 405 transforms the interlaced fields to progressivevideo, and denoising operations can then be performed on the progressivevideo.

After the appropriate inverse telecine or deinterlacing processing, atblock 408 the progressive video is processed for alias, suppressing andresampling (e.g., resizing).

After resampling, the progressive video then proceeds to block 410 wheredeblocker and deringing operations are performed. Two types ofartifacts, “blocking” and “ringing,” commonly occur in video compressionapplications. Blocking artifacts occur because compression algorithmsdivide each frame into blocks (e.g., 8×8 blocks). Each block isreconstructed with some small errors, and the errors at the edges of ablock often contrast with the errors at the edges of neighboring blocks,making block boundaries visible. In contrast, ringing artifacts appearas distortions around the edges of image features. Ringing artifactsoccur because the encoder discards too much information in quantizingthe high-frequency DCT coefficients. In some illustrative examples, bothdeblocking and deringing can use low-pass FIR (finite impulse response)filters to hide these visible artifacts.

After deblocking and deringing, the progressive video is processed by aGOP partitioner 412. GOP positioning can include detecting shot changes,generating complexity maps (e.g., temporal, spatial bandwidth maps), andadaptive GOP partitioning. Shot detection relates to determining when aframe in a group of pictures (GOP) exhibits data that indicates a scenechange has occurred. Scene change detection can be used for a videoencoder to determine a proper GOP length and insert I-frames based onthe GOP length, instead of insetting an I-frame at a fixed interval. Thepreprocessor 202 can also be configured to generate a bandwidth mapwhich can be used for encoding the multimedia data. In some aspects, acontent classification module located external to the preprocessorgenerates the bandwidth map instead. Adaptive GOP partitioning the canadaptively change the composition of a group of pictures coded together.Illustrative examples of is the operations shown in FIG. 4 are describedbelow.

Inverse Telecine

Inverse telecine processing is described below and an illustrativeexample of inverse telecine is provided in reference to FIGS. 4-16.Video compression gives best results when the properties of the sourceare known and used to select the ideally matching form of processing.Off-the-air video, for example, can originate in several ways. Broadcastvideo that is conventionally generated—in video cameras, broadcaststudios etc.—conforms in the United States to the NTSC standard.According to the standard, each frame is made up of two fields. Onefield consists of the odd lines, the other, the even lines. This may bereferred to as an “interlaced” format. While the frames are generated atapproximately 30 frames/sec, the fields are records of the televisioncamera's image that are 1/60 sec apart. Film on the other hand is shotat 24 frames/sec, each frame consisting of a complete image. This may bereferred to as a “progressive” format. For transmission in NTSCequipment, “progressive” video is converted into “interlaced” videoformat via a telecine process. In one aspect, further discussed below,the system advantageously determines when video has been telecined andperforms an appropriate transform to regenerate the original progressiveframes.

FIG. 5 shows the effect of telecining progressive frames that wereconverted to interlaced video. F₁, F₂, F₃, and F₄ are progressive images510 that are the input to a teleciner. The numbers “1” and “2” below therespective frames are indications of either odd or even fields. It isnoted that some fields are repeated in view of disparities amongst theframe rates. FIG. 5 also shows pull-down phases P₀, P₁, P₂, P₃, and P₄.The phase P₀ is marked by the first of two NTSC compatible frames 511which have identical first fields. The following four frames correspondto phases P₁, P₂, P₃, and P₄. Note that the frames marked by P₂ and P₃have identical second fields. Because film frame F₁ is scanned threetimes, two identical successive output NI′SC compatible first fields 511are formed. All NBC fields derived from Film frame F₁ are taken from thesame film image and therefore are taken at the same instant of time.Other NTSC frames derived from the film may have adjacent fields 1/24sec apart.

The phase detector 404 illustrated in FIG. 4 makes certain decisionsafter receipt of a video frame. These decisions include: (i) whether thepresent video from a telecine output and the 3:2 pull down phase is oneof the five phases P₀, P₁, P₂, P₃, and P₄ shown in definition 512 ofFIG. 5; and (ii) the video was generated as conventional NTSC—thatdecision is denoted as phase P₅.

These decisions appear as outputs of phase detector 401 shown in FIG. 4.The path from phase detector 401 labeled “YES” actuates the inversetelecine 406, indicating that it has been provided with the correct pulldown phase so that it can sort out the fields that were formed from thesame photographic image and combine them. The path from phase detector401 labeled “NO” similarly actuates the deinterlacer block 405 toseparate an apparent NTSC frame into fields for optimal processing.

FIG. 6 is a flowchart illustrating a process 600 of inverse telecining avideo stream. In one aspect, the process 600 is performed by the inversetelecine 301 of FIG. 3. Starting at a step 631, the inverse telecine 301determines a plurality of metrics based upon the received video. In thisaspect, four metrics are formed which are sums of differences betweenfields drawn from the same frame or adjacent frames. The four metricsare further assembled into a Euclidian measure of distance between thefour metrics derived from the received data and the most likely valuesof these metrics for each of the six hypothesized phases. The Euclideansums are called branch information; for each received frame there aresix such quantities. Each hypothesized phase has a successor phasewhich, in the case of the possible pull down phases, changes with eachreceived frame.

The possible paths of transitions are shown in FIG. 7 and denoted by767. There are six such paths. The decision process maintains sixmeasures equivalent to the sum of Euclidean distances for each path ofhypothesized phases. To make the procedure responsive to changedconditions each Euclidean distance in the sum is diminished as it getsolder. The phase track whose sum of Euclidean distances is smallest isdeemed to be the operative one. The current phase of this track iscalled the “applicable phase,” Inverse telecining based on the phaseselected, so long as it is not P₅, can now take place. If P₅ is selectedthen the current frame is deinterlaced using the deinterlacer at block405 (FIG. 4). In summary, the applicable phase is either utilized as thecurrent pull down phase, or as an indicator to command the deinterlaceof a frame that has been estimated to have a valid NTSC format.

For every frame received from the video input, a new value for each offour metrics is computed. These are defined as:SAD_(FS)=Σ|Current Field One Value(i,j)−Previous Field OneValue(i,j)|  (1)SAD_(SS)=Σ|Current Field Two Value(i,j)−Previous Field TwoValue(i,j)|  (2)SAD_(PO)=Σ|Current Field One Value(i,j)−Previous Field TwoValue(i,j)|  (3)SAD_(CO)=Σ|Current Field One Value(i,j)−Current Field TwoValue(i,j)|  (4)

The term SAD is an abbreviation of the term “summed absolutedifferences.” The fields which are differenced to form the metrics aregraphically shown m FIG. 8. The subscript refers to the field number;the letter denotes either Previous (=P) or Current (=C). The brackets inFIG. 8 refers to the pair-wise differencing of the fields. SAD_(FS)refers to differences between the field one of the current frame,labeled C₁, and field one of the previous frame, labeled P₁, which arespanned by a bracket labeled FS in definition provided in FIG. 8;SAD_(SS) refers to differences between the field two of the currentframe, labeled C₂, and field two of the previous frame, labeled P₂,which are both spanned by a bracket labeled SS; SAD_(CO) refers todifferences between field 2 of the current frame labeled C₂ and fieldone of the current frame, labeled C₁, which is spanned by a bracketlabeled CO; and SAD_(PO) refers to differences between field one of thecurrent frame and field 2 of the previous frame, which are both spannedby a bracket labeled PO.

The computational load to evaluate each SAD is described below. Thereare approximately 480 active horizontal lines in conventional NTSC. Forthe resolution to be the same in the horizontal direction, with a 4:3aspect ratio, there should be 480×4/3=640 equivalent vertical lines, ordegrees of freedom. The video format of 640×480 pixels is one of theformats accepted by the Advanced Television Standards Committee. Thus,every 1/30 of a second, the duration of a frame, 640×480=307,200 newpixels are generated. New data is generated at a rate of 9.2×10⁶pixels/sec, implying that the hardware or software running this systemprocesses data at approximately a 10 MB rate or more. This is one of thehigh speed portions of the system. It can be implemented by hardware,software, firmware, middleware, microcode, or any combination thereof.The SAD calculator could be a standalone component, incorporated ashardware, firmware, middleware in a component of another device, or beimplemented in microcode or software that is executed on the processor,or a combination thereof. When implemented in software, firmware,middleware or microcode, the program code or code segments that performthe calculation may be stored in a machine readable medium such as astorage medium. A code segment may represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents.

Flowchart 900 in FIG. 9 makes explicit the relationships in FIG. 8 andis a graphical representation of Equations 1-4. It shows storagelocations 941, 942, 943, and 944 into which are kept the most recentvalues of SAD_(FS), SAD_(CO), SAD_(SS) and SAD_(PO) respectively. Theseare each generated by four sum of absolute differences calculators 940,which process the luminance values of previous first field data 931,luminance values of current first field data 932, luminance values ofcurrent second field data 933 and luminance values of the previoussecond field data 934. In the summations that define the metrics, theterm “value(i,j)” is meant to be the value of the luminance at positioni,j, the summation being over all active pixels, though summing over thea meaningful subset of active pixels is not excluded.

Flowchart 100 in FIG. 10 is a detailed flowchart illustrating theprocess for detecting telecined video and inverting it to recover to theoriginal scanned film image. In step 1030 the metrics defined in FIG. 9are evaluated. Continuing to step 1083, lower envelope values of thefour metrics are found. A lower envelope of a SAD metric is adynamically determined quantity that is the highest numerical floorbelow which the SAD does not penetrate. Continuing to step 1085 branchinformation quantities defined below in Equations 5-10 are determined,which can use the previously determined metrics, the lower envelopevalues and an experimentally determined constant A. Since the successivevalues of the phase may be inconsistent, a quantity Δ is determined toreduce this apparent instability in step 1087. The phase is deemedconsistent when the sequence of phase decisions is, consistent with themodel of the problem shown in FIG. 7. Following that step, the processproceeds to step 1089 to calculate the decision variables using thecurrent value of Δ. Decision variables calculator 1089 evaluatesdecision variables using all the information generated in the blocks of1080 that led to it. Steps 1030, 1083, 1085, 1087, and 1089 are anexpansion of metrics determination 651 in FIG. 6. From these variables,the applicable phase is found by phase selector 1090. Decision step 1091uses the applicable phase to either invert the telecined video ordeinterlace it as shown. It is a more explicit statement of theoperation of phase detector 404 in FIG. 4. In one aspect the processingof FIG. 10 is performed by the phase detector 404 of FIG. 4. Starting atstep 1030, detector 404 determines a plurality of metrics by the processdescribed above with reference to FIG. 8, and continues through steps1083, 1085, 1087, 1089, 1090, and 1091.

Flowchart 1000 illustrates a process for estimating the current phase.The flowchart at a step 1083 describes the use of the determined metricsand lower envelope values to compute branch information. The branchinformation may be recognized as the Euclidean distances discussedearlier. Exemplary equations that may be used to generate the branchinformation are Equations 5-10 below. The Branch Info quantities arecomputed in block 1209 of FIG. 12.

The processed video data can be stored in a storage medium which caninclude, for example, a chip configured storage medium (e.g., ROM, RAM)or a disc-type storage medium (e.g., magnetic or optical) connected toprocessor. In some aspects, the inverse telecine 406 and thedeinterlacer 405 can each contain part or all of the storage medium. Thebranch information quantities are defined by the following equations.Branch Info(0)=(SAD_(FS) −H _(S))²+(SAD_(SS) −H _(S))²+(SAD_(PO) −H_(P))²+(SAD_(CO) −L _(C))²  (5)Branch Info(1)=(SAD_(FS) −L _(S))²+(SAD_(SS) −H _(S))²+(SAD_(PO) −L_(P))²+(SAD_(Co) −H _(C))²  (6)Branch Info(2)=(SAD_(FS) −H _(S))²+(SAD_(SS) −H _(S))²+(SAD_(PO) −L_(P))²+(SAD_(Co) −H _(C))²  (7)Branch Info(3)=(SAD_(FS) −H _(S))²+(SAD_(SS) −L _(S))²+(SAD_(PO) −L_(P))²+(SAD_(CO) −L _(C))²  (8)Branch Info(4)=(SAD_(FS) −H _(S))²+(SAD_(SS) −H _(S))²+(SAD_(PO) −H_(P))²+(SAD_(CO) −L _(C))²  (9)Branch Info(5)=(SAD_(FS) −L _(S))²+(SAD_(SS) −L _(S))²+(SAD_(PO) −L_(P))²+(SAD_(CO) −L _(C))²  (10)

The fine detail of the branch computation is shown in branch informationcalculator 1209 in FIG. 12. As shown in calculator 1209 developing thebranch information uses the quantities L_(S), the lower envelope valueof SAD_(FS) and SAD_(SS), L_(P), the lower envelope value of SAD_(PO),and L_(C), the lower envelope value of SAD_(CO). The lower envelopes areused as distance offsets in the branch information calculations, eitheralone or in conjunction with a predetermined constant A to create H_(S),H_(P) and H_(C). Their values are kept up to date in lower envelopetrackers described below. The H offsets are defined to be:H _(S) =L _(S) +A  (11)H _(PO) =L _(P) +A  (12)H _(C) =L _(C) +A  (13)

A process of tracking the values of L_(S), L_(P), and L_(C) is presentedin FIGS. 13A, 13B, and 13C. Consider, for example, the trackingalgorithm for L_(P) 1300 shown at the top of FIG. 11A. The metricSAD_(PO) is compared with the current value of L_(P) plus a thresholdT_(P) in comparator 1305. If it exceeds it, the current value of L_(P)is unchanged as shown in block 1315. If it does not, the new value ofL_(P) becomes a linear combination of SAD_(PO) and L_(P) as seen inblock 1313. In another aspect for block 1315 the new value of L_(P) isL_(P)+T_(P).

The quantities L_(S) and L_(C) in FIGS. 13B and 13C are similarlycomputed. Processing blocks in FIGS. 13A, 13B, and 13C which have thesame function are numbered identically but given primes (′ or ″) to showthat they operate on a different set of variables. For example, when alinear combination of the SAD_(PO) and L_(C) are formed, that operationis shown in block 1313′. As is the case for L_(P), another aspect for1315′ would replace L_(C) by L_(C)+T_(C).

In the case of L_(S), however, the algorithm in FIG. 13B processesSAD_(FS) and SAD_(SS) alternately, in turn labeling each X, since thislower envelope applies to both variables. The alternation of SAD_(FS)and SAD_(SS) values takes place when the current value of SAD_(FS) inblock 1308 is read into the location for X in block 1303, followed bythe current value of SAD_(SS) in 1307 being read into the location for Xin block 1302. As is the case for L_(P), another aspect for 1315″ wouldreplace L_(S) by L_(S)+T_(S). The quantity A and the threshold valuesused in testing the current lower envelope values are predetermined byexperiment.

FIG. 11 is a flowchart illustrating an exemplary, process for performingstep 1089 of FIG. 10. FIG. 11 generally shows a process for updating thedecision variables. There the six decision variables (corresponding tothe six possible decisions) are updated with new information derivedfrom the metrics. The decision variables are found as follows:D ₀ =αD ₄+Branch Info(0)  (14)D ₁ =αD ₀+Branch Info(1)  (15)D ₂ =αD ₁+Branch Info(2)  (16)D ₃ =αD ₂+Branch Info(3)  (17)D ₄ =αD ₃+Branch Info(4)  (18)D ₅ =αD ₅+Branch Info(5)  (19)

The quantity α is less than unity and limits the dependence of thedecision variables on their past values; use of α is equivalent todiminishing the effect of each Euclidean distance as its data ages. Inflowchart 1162 the decision variables to be updated are listed on theleft as available on lines 1101, 1102, 1103, 1104, 1105, and 1106. Eachof the decision variables on one of the phase transition paths is thenmultiplied by α, a number less than one in one of the blocks 1100; thenthe attenuated value of the old decision variable is added to thecurrent value of the branch info variable indexed by the next phase onthe phase transition path that the attenuated decision variable was on.This takes place in block 1110. Variable D₅ is offset by a quantity Δ inblock 1193; Δ is computed in block 1112. As described below, thequantity is chosen to reduce an inconsistency in the sequence of phasesdetermined by this system. The smallest decision variable is found inblock 1120.

In summary, new information specific to each decision is added to theappropriate decision variable's previous value that has been multipliedby α, to get the current decision variable's value. A new decision canbe made when new metrics are in hand; therefore this technique iscapable of making a new decision upon receipt of fields 1 and 2 of everyframe. These decision variables are the sums of Euclidean distancesreferred to earlier.

The applicable phase is selected to be the one having the subscript ofthe smallest decision variable. A decision based on the decisionvariables is made explicitly in block 1090 of FIG. 10. Certain decisionsare allowed in decision space. As described in block 1091, thesedecisions are: (i) The applicable phase in not P₅—inverse telecine thevideo and (ii) the applicable phase is P₅—deinterlace the video.

There may be occasional errors in a coherent string of decisions,because the metrics are drawn from video, which is inherently variable.This technique detects phase sequences that are inconsistent with FIG.7. Its operation is outlined in FIG. 14. The algorithm 1400 stores thesubscript of the present phase decision (=x) in block 1403 and thesubscript of the previous phase decision (=y) in block 1406. In block1410, if x=y=5 is tested; in block 1411 the following values are tested:

ifx=1,y=0; orx=2,y=1; orx=3,y=2; orx=4,y=3; orx=0,y=4.If either of the two tests is affirmative, the decisions are declared tobe consistent in block 1420. If neither test is affirmative, an offset,shown in block 1193 of FIG. 11 is computed in FIG. 15 and added to D₅,the decision variable associated with P₅.

The modification to D₅ also appears in FIG. 15 as part of process 1500,which provides corrective action to inconsistencies in a sequence ofphases. Suppose the consistency test in block 1510 in flowchart 1500 hasfailed. Proceeding along the “No” branch that leads from block 1510, thenext test in block 1514 is whether D₅>D_(i) for all i<5, oralternatively is at least one of the variables, D_(i), for i<5, biggerthan D₅. If the first case is valid, a parameter δ, whose initial valueis δ₀, is changed to 3δ₀ in block 1516. If the second, case is valid,then δ is changed to 4δ₀ in block 1517. In block 152B, the value of Δ isupdated to be Δ_(B), whereΔ_(B)=max(Δ−δ,−40δ₀)  (20)

Returning again to block 15210, assume that the string of decisions isjudged to be consistent. The parameter δ is changed to δ₊ in block15215, defined byδ₊=max(2δ,16δ₀)  (21)

The new value of δ is inserted into Δ_(A), the updating relationship forΔ in block 152A. This isΔ_(A)=max(Δ+δ,40δ₀)  (22)Then the updated value of Δ is added to decision variable D₅ in block1593.

FIG. 16 shows how the inverse telecine process proceeds once the pulldown phase is determined. With this information, fields 1605 and 1605′are identified as representing the same field of video. The two fieldsare averaged together, and combined with field 1606 to reconstruct frame1620. The reconstructed frame is 1620′. A similar process wouldreconstruct frame 1622. Fields derived from frames 1621 and 1623 are notduplicated. These frames are reconstructed by reweaving their first andsecond fields together.

In the aspect described above, every time a new frame is received fournew values of metrics are found and a six fold set of hypotheses istested using newly computed decision variables. Other processingstructures could be adapted to compute the decision variables. A Viterbidecoder adds the metrics of the branches that make up the paths togetherto form the path metric. The decision variables defined here are formedby a similar rule: each is the “leaky” sum of new information variables.(In a leaky summation the previous value of a decision variable ismultiplied by a number less than unity before new information data isadded to it.) A Viterbi decoder structure could be modified to supportthe operation of this procedure.

While the present aspect is described in terms of processingconventional video in which a new frame appears every 1/30 second, it isnoted that this process may be applied to frames which are recorded andprocessed backwards in time. The decision space remains the same, butthere are minor changes that reflect the time reversal of the sequenceof input frames. For example, a string of coherent telecine decisionsfrom the time-reversed mode (shown here)

-   -   P₄ P₃ P₂ P₁ P₀        would also be reversed in time.

Using this variation on the first aspect would allow the decisionprocess two tries—one going forward in time, the other backward—atmaking a successful decision. While the two tries are not independent,they are different in that each try would process the metrics in adifferent order.

This idea could be applied in conjunction with a buffer maintained tostore future video frames that may require additional processing. If avideo segment is found to give unacceptably inconsistent results in theforward direction of processing, the procedure would draw future framesfrom the buffer and attempt to get over the difficult stretch of videoby processing frames in the reverse direction.

The processing of video described in this patent can also be applied to,video in the PAL format.

Deinterlacer

“Deinterlacer” as used herein is a broad term that can be used todescribe a deinterlacing system, device, or process (including forexample, software, firmware, or hardware configured to perform aprocess) that processes, in whole or in significant part, interlacedmultimedia data to form progressive multimedia data.

Broadcast video that is conventionally generated—in video cameras,broadcast, studios etc.—conforms in the United States to the NTSCstandard. A common way to compress video is to interlace it. Ininterlaced data each frame is made up of one of two fields. One fieldconsists of the odd lines of the frame, the other, the even lines. Whilethe frames are generated at approximately 30 frames/sec, the fields arerecords of the television camera's image that are 1/60 sec apart. Eachframe of an interlaced video signal shows every other horizontal line ofthe image. As the frames are projected on the screen, the video signalalternates between showing even and odd lines. When this is done fastenough, e.g., around 60 frames per second, the video image looks smoothto the human eye.

Interlacing has been used for decades in analog television broadcaststhat are based on the NTSC (U.S.) and PAL (Europe) formats. Because onlyhalf tire image is sent with each frame, interlaced video uses roughlyhalf the bandwidth than it would sending the entire picture. Theeventual display format of the video internal to the terminals 16 is notnecessarily NTSC compatible and cannot readily display interlaced data.Instead, modern pixel-based displays (e.g., LCD, DLP, LCOS, plasma,etc.) are progressive scan and display progressively scanned videosources (whereas many older video devices use the older interlaced scantechnology). Examples of some commonly used deinterlacing algorithms aredescribed in “Scan rate up-conversion using adaptive weighted medianfiltering,” P. Haavisto, J. Juhola, and Y. Neuvo, Signal Processing ofHDTV II, pp. 703-710, 1990, and “Deinterlacing of HDTV Images forMultimedia Applications,” R. Simonetti, S. Carrato, G. Ramponi, and A.Polo Filisan, in Signal Processing of HDTV IV, pp. 765-772, 1993.

Described below are examples of deinterlacing aspects for systems andmethods that that can be used, solely or in combination, to improve theperformance of deinterlacing and which can be used in the deinterlacer405 (FIG. 4). Such aspects can include deinterlacing a selected frameusing spatio-temporal filtering to determine a first provisionaldeinterlaced frame, using bi-directional motion estimation and motioncompensation to determine a second provisional deinterlaced frame fromthe selected frame, and then combining the first and second provisionalframes to form a final progressive frame. The spatio-temporal filteringcan use a weighted median filter (“Wmed”), filter that can include ahorizontal edge detector that prevents blurring horizontal or nearhorizontal edges. Spatio-temporal filtering of previous and subsequentneighboring fields to a “current” field produces an intensitymotion-level map that categorizes portions of a selected frame intodifferent motion levels, for example, static, slow-motion, and fastmotion.

In some aspects, the intensity map is produced by Wmed filtering using afiltering aperture that includes pixels from five neighboring fields(two previous fields, the current field, and two next fields). The Wmedfiltering can determine forward, backward, and bidirectional static areadetection which can effectively handle scene changes and objectsappearing and disappearing. In various aspects, a Wmed filter can beutilized across one or more fields of the same parity in an inter-fieldfiltering mode, and switched to an intra-field filtering mode bytweaking threshold criteria. In some aspects, motion estimation andcompensation uses luma (intensity or brightness of the pixels) andchroma data (color information of the pixels) to improve deinterlacingregions of the selected frame where the brightness level is almostuniform but the color differs. A denoising filter can be used toincrease the accuracy of motion estimation. The denoising filter can beapplied to Wmed deinterlaced provisional frames to remove aliasartifacts generated by Wmed filtering. The deinterlacing methods andsystems described below produce good deinterlacing results and have arelatively low computational complexity that allow fast runningdeinterlacing implementations, making such implementations suitable fora wide variety of deinterlacing applications, including systems that areused to provide data to cell phones, computers and other types ofelectronic or communication devices utilizing a display.

The aspects of a deinterlacer and deinterlacing methods are describedherein with reference to various components, modules and/or steps thatare used to deinterlace multimedia data.

FIG. 17 is a block diagram illustrating one aspect of a deinterlacer1700 that can be used as the deinterlacer 405 in FIG. 4. Thedeinterlacer 1700 includes a spatial filter 1730 that spatially andtemporally (“spatio-temporal”) filters at least a portion of theinterlaced data and generates spatio-temporal information. For example,Wmed can be used in the spatial filter 1730. In some aspects thedeinterlacer 1700 also includes a denoising filter (not shown), forexample, a Weiner filter or a wavelet shrinkage filter. The deinterlacer1700 also includes a motion estimator 1732 which provides motionestimates and compensation of a selected frame of interlaced data andgenerates motion information. A combiner 1734 receives and combines thespatio-temporal information and the motion information to form aprogressive frame.

FIG. 18 is a block diagram of another deinterlacer 1800. A processor1836 in the deinterlacer 1800 includes a spatial filter module 1838, amotion estimation module 1840, and a combiner module 1842. Interlacedmultimedia data from an external source 48 can be provided to acommunications module 44 in the deinterlacer 1800. The deinterlacer, andcomponents or steps thereof, can be implemented by hardware, software,firmware, middleware, microcode, or any combination thereof. Forexample, a deinterlacer may be a standalone component, incorporated ashardware, firmware, middleware in a component of another device, or beimplemented in microcode or software that is executed on the processor,or a combination thereof. When implemented in software, firmware,middleware or microcode, the program code or code segments that performthe deinterlacer tasks may be stored in a machine readable medium suchas a storage medium. A code segment may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a class, or any combination of instructions, datastructures, or program statements. A code segment may be coupled toanother code segment or a hardware circuit by passing and/or receivinginformation, data, arguments, parameters, or memory contents.

The received interlaced data can be stored in the deinterlacer 1800 in astorage medium 1846 which can include, for example, a chip configuredstorage medium (e.g., ROM, RAM) or a disc-type storage medium (e.g.,magnetic or optical) connected to the processor 1836. In some aspects,the processor 1836 can contain part or all of the storage medium. Theprocessor 1836 is configured to process the interlaced multimedia datato form progressive frames which are then provided to another device orprocess.

Traditional analog video devices like televisions render video in aninterlaced manner, i.e., such devices transmit even-numbered scan lines(even field), and odd-numbered scan lines (odd field). From the signalsampling point of view, this is equivalent to a spatio-temporalsubsampling in a pattern described by:

$\begin{matrix}{{F\left( {x,y,n} \right)} = \left\{ \begin{matrix}{{\Theta\left( {x,y,n} \right)},{{{if}\mspace{14mu} y\;{mod}\; 2} = {0\mspace{14mu}{for}\mspace{14mu}{even}\mspace{14mu}{fields}}},} \\{{\Theta\left( {x,y,n} \right)},{{{if}\mspace{14mu} y\;{mod}\; 2} = {1\mspace{14mu}{for}\mspace{14mu}{odd}\mspace{14mu}{fields}}},} \\{{Erasure},{otherwise},}\end{matrix} \right.} & (23)\end{matrix}$where Θ stands for the original frame picture, F stands for theinterlaced field, and (x, y, n) represents the horizontal, vertical, andtemporal position of a pixel respectively.

Without loss of generality, it can be assumed n=0 is an even fieldthroughout this disclosure so that Equation 23 above is simplified as

$\begin{matrix}{{F\left( {x,y,n} \right)} = \left\{ \begin{matrix}{{\Theta\left( {x,y,n} \right)},{{{if}\mspace{14mu} y\;{mod}\; 2} = {n\;{mod}\; 2}},} \\{{Erasure},{otherwise},}\end{matrix} \right.} & (24)\end{matrix}$

Since decimation is not conducted in the horizontal dimension, thesub-sampling pattern can be depicted in the next n˜y coordinate. In FIG.19, both circles and stars represent positions where the originalfoil-frame picture has a sample pixel. The interfacing process decimatesthe star pixels, while leaving the circle pixels intact. It should benoted that we index vertical positions starting from zero, therefore theeven field is the top field, and the odd field is the bottom field.

The goal of a deinterlacer is to transform interlaced video (a sequenceof fields) into non-interlaced progressive frames (a sequence offrames). In other words, interpolate even and odd fields to “recover” orgenerate full-frame pictures. This can be represented by Equation 25:

$\begin{matrix}{{F_{a}\left( {x,y,n} \right)} = \left\{ \begin{matrix}{{F\left( {x,y,n} \right)},{{y\;{mod}\; 2} = {n\;{mod}\; 2}},} \\{{F_{i}\left( {x,y,n} \right)},{otherwise},}\end{matrix} \right.} & (25)\end{matrix}$where F_(i) represent deinterlacing results for missing pixels.

FIG. 20 is a block diagram illustrating certain aspects of an aspect ofa deinterlacer that uses Wmed filtering and motion estimation togenerate a progressive frame from interlaced multimedia data. The upperpart of FIG. 20 shows a motion intensity map 2052 that can be generatedusing information from a current field, two previous fields (PP Fieldand P Field), and two subsequent fields (Next Field and Next Nextfield). The motion intensity map 2052 categorizes, or partitions, thecurrent frame into two or more different motion levels, and can begenerated by spatio-temporal filtering, described in further detailhereinbelow. In some aspects, the motion intensity map 2052 is generatedto identify static areas, slow-motion areas, and fast-motion areas, asdescribed in reference to Equations 4-8 below. A spatio-temporal filter,e.g., Wmed filter 2054, filters the interlaced multimedia data usingcriteria based on the motion intensity map, and produces aspatio-temporal provisional deinterlaced frame. In some aspects, theWmed filtering process involves a horizontal a neighborhood of [−1, 1],a vertical neighborhood of [−3, 3], and a temporal neighborhood of fiveadjacent fields, which are represented by the five fields (PP Field, PField, Current Field, Next Field, Next Next Field) illustrated in FIG.20, with Z⁻¹ representing a delay of one field. Relative to the CurrentField, the Next Field and the P Field are non-parity fields and the PPField and the Next Next Field are parity fields. The “neighborhood” usedfor spatio-temporal filtering refers to the spatial and temporallocation of fields and pixels actually used during the filteringoperation, and can be illustrated as an “aperture” as shown, forexample, in FIGS. 21 and 22.

The deinterlacer can also include a denoiser (denoising filter) 2056.The denoiser 2056 is configured to filter the spatio-temporalprovisional deinterlaced frame generated by the Wmed filter 2056.Denoising the spatio-temporal provisional deinterlaced frame makes thesubsequent motion search process more accurate especially if the sourceinterlaced multimedia data sequence is contaminated by white noise. Itcan also at least partly remove alias between even and odd rows in aWmed picture. The denoiser 2056 can be implemented as a variety offilters including a wavelet shrinkage and wavelet Wiener filter baseddenoiser which are also described further hereinbelow.

The bottom part of FIG. 20 illustrates an aspect for determining motioninformation (e.g., motion vector candidates, motion estimation, motioncompensation) of interlaced multimedia data. In particular, FIG. 20illustrates a motion estimation and motion compensation scheme that isused to generate a motion compensated provisional progressive frame ofthe selected frame, and then combined with the Wmed provisional frame toform a resulting “final” progressive frame, shown as deinterlacedcurrent frame 2064. In some aspects, motion vector (“MV”) candidates (orestimates) of the interlaced multimedia data are provided to thedeinterlacer from external motion estimators and used to provide astarting point for bi-directional motion estimator and compensator(“ME/MC”) 2068. In some aspects, a MV candidate selector 2072 usespreviously determined MV's for neighboring blocks for MV candidates ofthe blocks being processed, such as the MVs of previous processedblocks, for example blocks in a deinterlaced previous frame 2070. Themotion compensation can be done bi-directional, based on the previousdeinterlaced frame 70 and a next (e.g., future) Wmed frame 2058. Acurrent Wmed frame 2060 and a motion compensated (“MC”) current frame2066 are merged, or combined, by a combiner 2062. A resultingdeinterlaced current frame 2064, now a progressive frame, is providedback to the ME/MC 2068 to be used as a deinterlaced previous frame 2070and also communicated external to the deinterlacer for furtherprocessing, e.g., compression and transmission to a display terminal.The various aspects shown in FIG. 20 are described in more detail below.

FIG. 25 illustrates a process 2500 for processing multimedia data toproduce a sequence of progressive frames from a sequence of interlacedframes. In one aspect, a progressive frame is produced by thedeinterlacer 405 illustrated in FIG. 4. At block 2502, process 2500(process “A”) generates spatio-temporal information for a selectedframe. Spatio-temporal information can include information used tocategorize the motion levels of the multimedia data and generate amotion intensity map, and includes the Wmed provisional deinterlacedframe and information used to generate the frame (e.g., information usedin Equations 26-33). This process can be performed by die Wmed filter2054, as illustrated in the upper portion of FIG. 20, and its associatedprocessing, which is described in further detail below. In process A,illustrated in FIG. 26, regions are classified into fields of differentmotion levels at block 2602, as further described below.

Next, at block 2504 (process “B), process 2500 generates motioncompensation information for a selected frame. In one aspect, thebi-directional motion estimator/motion compensator 2068, illustrated inthe lower portion of FIG. 20, can perform this process. The process 2500then proceeds to block 2506 where it deinterlaces fields of the selectedframe based on the spatio-temporal information and the motioncompensation information to form a progressive frame associated with theselected frame. This can be performed by the combiner 2062 illustratedin the lower portion of FIG. 20.

Motion Intensity Map

For each frame, a motion intensity 2052 map can be determined byprocessing pixels in a current field to determine areas of different“motion.” An illustrative aspect of determining a three category motionintensity map is described below with reference to FIGS. 21-24. Themotion intensity map designates areas of each frame as static areas,slow-motion areas, and fast motion areas based on comparing pixels insame-parity fields and different parity fields.

Static Areas

Determining static areas of the motion map can comprise processingpixels in a neighborhood of adjacent fields to determine if luminancedifferences of certain pixel(s) meet certain criteria. In some aspects,determining static areas of the motion map comprises processing pixelsin a neighborhood of five adjacent fields (a Current Field (C), twofields temporally before the current field, and two frames temporallyafter the Current Field) to determine if luminance differences ofcertain pixel(s) meet certain thresholds. These five fields areillustrated in FIG. 20 with Z⁻¹ representing a delay of one field. Inother words, the five adjacent would typically be displayed in such asequence with a Z⁻¹ time delay.

FIG. 21 illustrates an aperture identifying certain pixels of each ofthe five fields that can be used for the spatio-temporal filtering,according to some aspects. The aperture includes, from left to right,3×3 pixel groups of a Previous Previous Field (PP), a Previous Field(P), the Current Field (C), a Next Field (N), and a Next Next Field(NN). In some aspects, an area of the Current Field is considered staticin the motion map if it meets the criteria described in the Equations26-28, the pixel locations and corresponding fields being illustrated inFIG. 21:|L _(P) −L _(N) |<T ₁  (26)

and

$\begin{matrix}\left( {{{\frac{L_{BPP} - L_{B}}{2}} + {\frac{L_{EPP} - L_{E}}{2}}} < {T_{1}\mspace{11mu}\left( {{forward}\mspace{14mu}{static}} \right){or}}} \right. & (27) \\{\left. {{{\frac{L_{BNN} - L_{B}}{2}} + {\frac{L_{ENN} - L_{E}}{2}}} < {T_{1}\mspace{14mu}\left( {{backward}\mspace{14mu}{static}} \right)}} \right),} & (28)\end{matrix}$

where T₁ is a threshold,

L_(P) is the Luminance of a pixel P located in the P Field,

L_(N) is the luminance of a pixel N located in the N Field,

L_(B) is the Luminance of a pixel B located in the Current Field,

L_(E) is the Luminance of a pixel E located in the Current Field,

L_(BPP) is the Luminance of a pixel B_(PP) located in the PP Field,

L_(EPP) is the Luminance of a pixel E_(PP) located in the PP Field,

L_(BNN) is the luminance of a pixel B_(NN) located in the NN Field, and

L_(ENN) is the Luminance of a pixel E_(NN) located in the NN Field.

Threshold T₁ can be predetermined and set at a particular value,determined by a process other than deinterlacing and provided (forexample, as metadata for the video being deinterlaced) or it can bedynamically determined during deinterlacing.

The static area criteria described above in Equation 26, 27, and 28 usemore fields than conventional deinterlacing techniques for at least tworeasons. First, comparison between same-parity fields has lower aliasand phase-mismatch than comparison between different-parity fields.However, the least time difference (hence correlation) between the fieldbeing processed and its most adjacent same-parity field neighbors is twofields, larger than that from its different-parity field neighbors. Acombination of more reliable different-parity fields and lower-aliassame-parity fields can improve the accuracy of the static areadefection.

In addition, the five fields can be distributed symmetrically in thepast and in the future relative to a pixel X in the Current Frame C, asshown in FIG. 21. The static area can be sub-divided into threecategories: forward static (static relative to the previous frame),backward static (static relative to the next frame), or bi-directional(if both the forward and the backward criteria are satisfied). Thisfiner categorization of the static areas can improve performanceespecially at scene changes and object appearing/disappearing.

Slow-Motion Areas

An area of the motion-map can be considered a slow-motions area in themotion-map if the luminance values of certain pixels do not meet thecriteria to be designated a static area but meet criteria to bedesignated a slow-motion area. Equation 29 below defines criteria thatcan be used to determine a slow-motion area. Referring to FIG. 22, thelocations of pixels Ia, Ic, Ja, Jc, Ka, Kc, La, Lc, P and N identifiedin Equation 29 are shown in an aperture centered around pixel X. Theaperture includes a 3×7 pixel neighborhood of the Current Field (C) and3×5 neighborhoods of the Next Field (N) a Previous Field (P). Pixel X isconsidered to be part of a slow-motion area if it does not meet theabove-listed criteria for a static area and if pixels in the aperturemeet the following criteria shown in Equation 29:(|L _(Ia) −L _(Ic) |+|L _(Ja) −L _(Jc) |+|L _(Ja) −L _(Jc) |+|L _(Ka) −L_(Kc) |+|L _(La) −L _(Lc) |+|L _(P) −L _(N)|)/5<T ₂  (29)

where T₂ is a threshold, and

-   -   L_(Ia), L_(Ic), L_(Ja), L_(Jc), L_(Ja), L_(Jc), L_(Ka), L_(Kc),        L_(La), L_(Lc), L_(P), L_(N) are luminance values for pixels Ia,        Ic, Ja, Jc, Ka, Kc, La, Lc, P and N, respectively.

The threshold T₂ can also fee predetermined and set at a particularvalue, determined by a process other than deinterlacing and provided(for example, as metadata for the video being deinterlaced) or it can bedynamically determined during deinterlacing.

It should be noted that a filter can blur edges that are horizontal(e.g., more than 45° from vertically aligned) because of the angle ofits edge detection capability. For example, the edge detectioncapability of the aperture (filter) illustrated in FIG. 22 is affectedby the angle formed by pixel “A” and “F”, or “C” and “D”. Any edges morehorizontal than such an angle that will not be interpolated optimallyand hence staircase artifacts may appear at those edges. In someaspects, the slow-motion category can be divided into twosub-categories. “Horizontal Edge” and “otherwise” to account for thisedge detection effect. The slow-motion pixel can be categorized as aHorizontal Edge if the criteria in Equation 30, shown below, issatisfied, and to a so-called “Otherwise” category if the criteria inEquation 30 is not satisfied,|(LA+LB+LC)−(LD+LE+LF)|<T ₃  (30)where T₃ is a threshold, and LA, LB, LC, LD, LE, and LF are theluminance values of pixels A, B, C, D, E, and F.

Different interpolation methods can used for each of the Horizontal Edgeand the Otherwise category.

Fast-Motion Areas

If the criteria for a static area and the criteria for the slow-motionarea are not met, the pixel can be deemed to be in a fast-motion area.

Having categorized the pixels in a selected frame, process A (FIG. 26)then proceeds to block 2604 and generates a provisional deinterlacedframe based upon the motion intensity map. In this aspect, Wmed filter2054 (FIG. 20) filters the selected field and the necessary adjacentfields(s) to provide a candidate full-frame image F₀ which can bedefined as follows:

$\begin{matrix}{{F_{n}\left( {x,n} \right)} = \left\{ \begin{matrix}{F\left( {\overset{\_}{x},n} \right)} & \left( {{y\mspace{14mu}{mod}\mspace{14mu} 2} = {n\mspace{14mu}{mod}\mspace{14mu} 2}} \right) \\{{\frac{1}{2}\left( {{F\left( {\overset{\_}{x},{n - 1}} \right)} + {F\left( {\overset{\_}{x},{n + 1}} \right)}} \right)},} & \left( {{static}\mspace{14mu}{backward}\mspace{14mu}{and}\mspace{14mu}{forward}} \right) \\{F\left( {\overset{\_}{x},{n - 1}} \right)} & \left( {{static}\mspace{14mu}{forward}\mspace{14mu}{but}\mspace{14mu}{not}\mspace{14mu}{forward}} \right) \\{F\left( {\overset{\_}{x},{n + 1}} \right)} & \left( {{static}\mspace{14mu}{backward}\mspace{14mu}{but}\mspace{14mu}{not}\mspace{14mu}{backward}} \right) \\{{med}\left( {A,B,C,D,E,F} \right)} & \left( {{slow}\mspace{14mu}{motion}\mspace{14mu} w\text{/}o\mspace{14mu}{horizontal}\mspace{14mu}{edge}} \right) \\{{{med}\left( {{\alpha_{0}\frac{A + F}{2}},{\alpha_{1}\frac{B + E}{2}},{\alpha_{2}\frac{C + D}{2}},{\alpha_{3}\frac{G + H}{2}}} \right)}.} & \left( {{slow}\mspace{14mu}{motion}\mspace{14mu} w\text{/}\mspace{14mu}{horizontal}\mspace{14mu}{edge}} \right) \\{\frac{B + E}{2},} & \left( {{fast}\mspace{14mu}{motion}} \right)\end{matrix} \right.} & (31)\end{matrix}$where α_(i)(i=0, 1, 2, 3) are integer weights calculated as below:

$\begin{matrix}{\alpha_{1} = \left\{ \begin{matrix}2 & {{{if}\mspace{14mu}\beta_{1}} = {\min\left\{ {\beta_{0},\beta_{1},\beta_{2},\beta_{3}} \right\}}} \\{1,} & {otherwise}\end{matrix} \right.} & (32) \\{{\beta_{0} = \frac{A + F}{{A - F}}},{\beta_{1} = \frac{B + E}{{B - E}}},{\beta_{2} = \frac{C + D}{{C - D}}},{\beta_{3} = \frac{G + H}{{G - H}}}} & (33)\end{matrix}$

The Wmed filtered provisional deinterlaced frame is provided for furtherprocessing in conjunction with motion estimation and motion compensationprocessing, as illustrated in the lower portion of FIG. 20.

As described above and shown in Equation 31, the static interpolationcomprises inter-field interpolation and the slow-motion and fast-motioninterpolation comprises intra-field interpolation. In certain aspectswhere temporal (e.g., inter-field) interpolation of same parity fieldsis not desired, temporal interpolation can be “disabled” by setting thethreshold T₁ (Equations 4-6) to zero (T₁=0). Processing of the currentfield with temporal interpolation disabled results in categorizing noareas of the motion-level map as static, and the Wmed filter 2054 (FIG.20) uses the three fields illustrated in the aperture in FIG. 22 whichoperate on a current field and the two adjacent non-parity fields.

Denoising

In certain aspects, a denoiser can be used to remove noise from thecandidate Wmed frame before it is further processed using motioncompensation information. A denoiser can remove noise that is present inthe Wmed frame and retain the signal present regardless of the signal'sfrequency content. Various types of denoising filters can be used,including wavelet filters. Wavelets are a class of functions used tolocalize a given signal in both space and scaling domains. Thefundamental idea behind wavelets is to analyze the signal at differentscales or resolutions such that small changes in the waveletrepresentation produce a correspondingly small change in the originalsignal.

In some aspects, a denoising filter is based on an aspect of a (4, 2)bi-orthogonal cubic B-spline wavelet filter. One such filter can bedefined by the following forward and inverse transforms:

$\begin{matrix}{{{h(z)} = {\frac{3}{4} + {\frac{1}{2}\left( {z + z^{- 1}} \right)} + {\frac{1}{8}\left( {z + z^{- 2}} \right)\mspace{14mu}\left( {{forward}\mspace{14mu}{transform}} \right)}}}{and}} & (34) \\{{g(z)} = {{\frac{5}{4}z^{- 1}} - {\frac{5}{32}\left( {1 + z^{- 2}} \right)} - {\frac{3}{8}\left( {z + z^{- 3}} \right)} - {\frac{3}{32}\left( {z^{2} + z^{- 4}} \right)\mspace{14mu}\left( {{inverse}\mspace{14mu}{transform}} \right)}}} & (35)\end{matrix}$

Application of a denoising filter can increase the accuracy of motioncompensation in a noisy environment. Noise in the video sequence isassumed to be additive white Gaussian. The estimated variance of thenoise is denoted by σ. It can be estimated as the median absolutedeviation of the highest-frequency subband coefficients divided by0.6745. Implementations of such filters are described further in “Idealspatial adaptation by wavelet shrinkage,” D. L. Donoho and L. M.Johnstone, Biometrika, vol. 8, pp. 425-455, 1994, which is incorporatedby reference herein in its entirety.

A wavelet shrinkage or a wavelet Wiener filter can be also be applied asthe denoiser. Wavelet shrinkage denoising can involve shrinking in thewavelet transform domain, and typically comprises three steps: a linearforward wavelet transform, a nonlinear shrinkage denoising, and a linearinverse wavelet transform. The Wiener filter is a MSE-optimal linearfilter which can be used to improve images degraded by additive noiseand blurring. Such filters are generally known in the art and aredescribed, for example, in “Ideal spatial adaptation by waveletshrinkage,” referenced above, and by S. P. Ghael A. M. Sayeed, and R. G.Baraniuk, “Improvement Wavelet denoising via empirical Wienerfiltering,” Proceedings of SPIE, vol 3169, pp. 389-399, San Diego, July1997.

Motion Compensation

Referring to NG. 27, at block 2702 process performs bi-directionalmotion estimation, and then at block 104 uses motion estimates toperform motion compensation, which is illustrated in FIG. 20, anddescribed in an illustrative aspect below. There is a one field “lag”between the Wined filter and the motion-compensation based deinterlacer.Motion compensation information for the “missing” data (the non-originalrows of pixel data) of the Current Field “C” is being predicted frominformation in both the previous frame “P” and the next frame “N” asshown in FIG. 23. In the Current Field (FIG. 23) solid tines representrows where original pixel data exist and dashed lines represent rowswhere Wined-interpolated pixel data exist. In certain aspects, motioncompensation is performed in a 4-row by 8-column pixel neighborhood.However, this pixel neighborhood is an example for purposes ofexplanation, and it will be apparent to those skilled in the art thatmotion compensation may be performed in other aspects based on a pixelneighborhood comprising a different number rows and a different numberof columns, the choice of which can be based on many factors including,for example, computational speed, available processing power, orcharacteristics of the multimedia data being deinterlaced. Because theCurrent Field only has half of the rows, the four rows to be matchedactually correspond to an 8-pixel by 8-pixel area.

Referring to FIG. 20, the bi-directional ME/MC 2068 can Use sum ofsquared errors (SSE) can be used to measure the similarity between apredicting block and a predicted block for the Wmed current frame 2060relative to the Wmed next frame 2058 and the deinterlaced current frame2070. The generation of the motion compensated current frame 2066 thenuses pixel information from the most similar matching blocks to fill inthe missing data between the original pixel lines. In some aspects, thebi-directional ME/MC 2068 biases or gives more weight to the pixelinformation from the deinterlaced previous frame 2070 informationbecause it was generated by motion compensation information and Wmedinformation, while the Wmed next frame 2058 is only deinterlaced byspatio-temporal filtering.

In some aspects, to improve matching performance in regions of fieldsthat have similar-luma regions but different-chroma regions, a metriccan be used that includes the contribution of pixel values of one ormore luma group of pixels (e.g., one 4-row by 8-column luma block) andone or more chroma group of pixels (e.g., two 2-row by 4-column chromablocks U and V). Such approaches effectively reduces mismatch at colorsensitive regions.

Motion Vectors (MVs) have granularity of ½ pixels in the verticaldimension, and either ½ or ¼ pixels in the horizontal dimension. Toobtain fractional-pixel samples, interpolation filters can be used. Forexample, some filters that can be used to obtain half-pixel samplesinclude a bilinear filter (1, 1), an interpolation filter recommended byH.263/AVC: (1, −5, 20, 20, −5, 1), and a six-tap Hamming windowed sinefunction filter (3, −21, 147, 147, −21, 3). ¼-pixel samples can begenerated from full and half pixel sample by applying a bilinear filter.

In some aspects, motion compensation can use various types of searchingprocesses to match data (e.g., depicting an object) at a certainlocation of a current frame to corresponding data at a differentlocation in another frame (e.g., a next frame or a previous frame), thedifference in location within the respective frames indicating theobject's motion. For example, the searching processes use a full motionsearch which may cover a larger search area or a fast motion searchwhich can use fewer pixels, and/or the selected pixels used in thesearch pattern can have a particular shape, e.g., a diamond shape. Forfast motion searches, the search areas can be centered around motionestimates, or motion candidates, which can be used as a starting pointfor searching the adjacent frames. In some aspects, MV candidates can begenerated from external motion estimators and provided to thedeinterlacer. Motion vectors of a macroblock from a correspondingneighborhood in a previously motion compensated adjacent frame can alsobe used as a motion estimate. In some aspects, MV candidates can begenerated from searching a neighborhood of macroblocks (e.g., a3-macroblock by 3-macroblock) of the corresponding previous and nextframes.

FIG. 24 illustrates an example of two MV maps, MV_(P) and MV_(N), thatcould be generated during motion estimation/compensation by searching aneighborhood of the previous frame and the next frame, as show in FIG.23. In both MV_(P) and MV_(N) the block to be processed to determinemotion information is the center block denoted by “X.” In both MV_(P)and MV_(N), there are nine MV candidates that can be used during motionestimation of the current block X being processed. In this example, fourof the MV candidates exist in the same field from earlier performedmotion searches and are depicted by the lighter-colored blocks in MV_(P)and MV_(N) (FIG. 24). Five other MV candidates, depicted by thedarker-colored blocks, were copied from the motion information (or maps)of the previously processed frame.

After motion estimation/compensation is completed, two interpolationresults may result for the missing rows (denoted by the dashed lines inFIG. 23): one interpolation result generated by the Wmed filter (WmedCurrent Frame 2060 FIG. 20) and one interpolation result generated bymotion estimation processing of the motion compensator (MC Current Frame2066). A combiner 2062 typically merges the Wmed Current Frame 2060 andthe MC Current Frame 2066 by using at least a portion of the WmedCurrent Frame 2060 and the MC Current Frame 2066 to generate a CurrentDeinterlaced Frame 2064. However, under certain conditions, the combiner2062 may generate a Current Deinterlaced frame using only one of theCurrent Frame 2060 or the MC Current Frame 2066. In one example, thecombiner 2062 merges the Wmed Current Frame 2060 and the MC CurrentFrame 2066 to generate a deinterlaced output signal as shown in Equation36:

$\begin{matrix}{{F_{o}\left( {x,n} \right)} = \left\{ \begin{matrix}, & \left( {{y\mspace{14mu}{mod}\mspace{11mu} 2} = {n\mspace{14mu}{mod}\mspace{14mu} 2}} \right) \\ & {({otherwise}).}\end{matrix} \right.} & (36)\end{matrix}$where F( x,n) is used for the luminance value in field n₁ at positionx=(x, y)^(t) with ^(t) for transpose. Using a clip function defined asclip(0,1,a)=0, if (a<0); 1, if (a>1); a, otherwise  (37)k₁ can be calculated as:k ₁=clip(0,C ₁√{square root over (Diff)})  (38)where C₁ is a robustness parameter, and Diff is the luma differencebetween the predicting frame pixel and the available pixel in thepredicted frame (taken from the existing field). By appropriatelychoosing C1, it is possible to tune the relative importance of the meansquare error. k₂ can be calculated as shown in Equation 39:

$\begin{matrix}{k_{2} = {1 - {{clip}\left( {0,1,{\left( {1 - k_{1}} \right)\frac{{{{F_{Wmed}\left( {{\overset{\rightharpoonup}{x} - {\overset{\rightharpoonup}{y}}_{u}},n} \right)} - {F_{MC}\left( {{\overset{\rightharpoonup}{x} - {\overset{\rightharpoonup}{y}}_{u} - \overset{\rightharpoonup}{D}},{n - 1}} \right)}}} + {\delta}}{{{{F_{Wmed}\left( {\overset{\rightharpoonup}{x},n} \right)} - {F_{MC}\left( {{\overset{\rightharpoonup}{x} - \overset{\rightharpoonup}{D}},{n - 1}} \right)}}} + \delta}}} \right)}}} & (39)\end{matrix}$where {right arrow over (x)}=(x, y), y _(u)=(0,1), {right arrow over(D)} is the motion vector, δ is a small constant to prevent division byzero. Deinterlacing using clipping functions for filtering is furtherdescribed in “De-interlacing of video data,” G. D. Haan and E. B.Bellers, IEEE Transactions on Consumer Electronics, Vol. 43, No. 3, pp.819-825, 1997, which is incorporated herein in its entirety.

In some aspects, the combiner 2062 can be configured to try and maintainthe following equation to achieve a high PSNR and robust results:|F _(O)( x,n)−F _(Wmed)( x,n)|=|F _(O)( x− y _(u) ,n)−F _(Wmed)( x− y_(u) ,n)|  (40)

It is possible to decouple deinterlacing prediction schemes comprisinginter-field interpolation from intra-field interpolation with a Wmed+MCdeinterlacing scheme. In other words, the spatio-temporal Wmed filteringcan be used mainly for intra-field interpolation purposes, whileinter-field interpolation can be performed during motion compensation.This reduces the peak signal-to-noise ratio of the Wmed result, but thevisual quality after motion compensation is applied is more pleasing,because bad pixels from inaccurate inter-field prediction mode decisionswill be removed from the Wmed filtering process.

Chroma handling can be consistent with the collocated luma handling. Interms of motion map generation, the motion level of a chroma pixel isobtained by observing the motion level of its four collocated lumapixels. The operation can be based on voting (chroma motion levelborrows the dominant luma motion level). However, we propose to use aconservative approach as follows. If any one of the four luma pixels hasa fast motion level, the chroma motion level shall be fast-motion; otherwise, if any one of the four luma pixels has a slow motion level, thechroma motion level shall be slow-motion; otherwise the chroma motionlevel is static. The conservative approach may not achieve the highestPSNR, but it avoids the risk of using INTER prediction wherever there isambiguity in chroma motion level.

Multimedia data sequences were deinterlaced using the described Wmedalgorithm described alone and the combined Wmed and motion compensatedalgorithm described herein. The same multimedia data sequences were alsodeinterlaced using a pixel blending (or averaging) algorithm and a“no-deinterlacing” case where the fields were merely combined withoutany interpolation or blending. The resulting frames were analyzed todetermine the PSNR and is shown in the following table:

PSNR (dB) no sequence deinterlacing blending Wmed Wmed + MC soccer8.955194 11.38215 19.26221 19.50528 city 11.64183 12.93981 15.0330315.09859 crew 13.32435 15.66387 22.36501 22.58777

Even though titers is only marginal PSNR improvement by deinterlacingusing the MC in addition to Wmed, the visual quality of the deinterlacedimage produced by combining the Wmed and MC interpolation results ismore visually pleasing to because as mentioned above, combining the Wmedresults and the MC results suppresses alias and noise between even andodd fields.

In some resampling aspects, a poly-phase resampler is implemented forpicture size resizing. In one example of downsampling, the ratio betweenthe original and the resized picture can be p/g, where p and q arerelatively prime integers. The total number of phases is p. The cutofffrequency of the poly-phase filter in some aspects is 0.6 for resizingfactors around 0.5. The cutoff frequency does not exactly match theresizing ratio in order to boost the high-frequency response of theresized sequence. This inevitably allows some aliasing. However, it iswell-known that human eyes prefer sharp but a little aliased pictures toblurry and alias-free pictures.

FIG. 42 illustrates an example of poly-phase resampling, showing thephases if the resizing ration is ¾. The cutoff frequency illustrated inFIG. 42 is ¾ also. Original pixels are illustrated in the above FIG. 42with vertical axes. A sine function is also drawn centered around theaxes to represent the filter waveform. Because we choose the cutofffrequency to be exactly the same as the resampling ration, the zeros ofthe sine function overlap the position of the pixels after resizing,illustrated in FIG. 42 with crosses. To find a pixel value afterresizing, the contribution can be summed up from the original pixels asshown in the following equation:

$\begin{matrix}{{y(x)} = {\sum\limits_{i = {- \infty}}^{\infty}{{u(i)} \times \sin\;{c\left( {\pi\;{f_{c}\left( {i - x} \right)}} \right)}}}} & (41)\end{matrix}$where f_(c) is the cutoff frequency. The above 1-D poly-phase filter canbe applied to both the horizontal dimension and the vertical dimension.

Another aspect of resampling (resizing) is accounting for overscan. Inan NTSC television signal, an image has 486 scan lines, and in digitalvideo could have 720 pixels on each scan line. However, not all of theentire image is visible on the television due to mismatches between thesize and the screen format. The part of the image that is not visible iscalled overscan.

To help broadcasters put useful information in the area visible by asmany televisions as possible, the Society of Motion Picture & TelevisionEngineers (SMPTE) defined specific sizes of the action frame called thesafe action area and the safe title area. See SMPTE recommended practiceRP 27.3-1989 on Specifications for Safe Action and Safe Title Areas TestPattern for Television Systems. The safe action area is defined by theSMPTE as the area in which “all significant action must take place.” Thesafe title area is defined as the area where “all the useful informationcan be confined to ensure visibility on the majority of home televisionreceivers.” For example, as illustrated in FIG. 43, the safe action area4310 occupies the center 90% of the screen, giving a 5% border allaround. The safe title area 4305 occupies the center 80% of the screen,giving a 10% border. FIG.

Referring now to FIG. 44, because the safe title area is so small, toadd more contents in the image, some broadcasts will include text in thesafe action area, which is inside the white rectangular window 4415.Usually black borders may be seen in the overscan. For example, in FIG.44, black borders appear at the upper side 4420 and lower side 4425 ofthe image. These black borders can be removed in the overscan, becauseH.264 video uses boundary extension in motion estimation. Extended blackborders can increase the residual. Conservatively, we can cut theboundary by 2%, and then do the resizing. The filters for resizing canbe generated accordingly. Truncation is performed to remove the overscanbefore poly-phase downsampling.

Deblocking/Deringing

In one example of deblocking processing, a deblocking filter can beapplied to all the 4×4 block edges of a frame, except edges at theboundary of the frame and any edges for which the deblocking filterprocess is disabled. This filtering process shall be performed on amacroblock basis after the completion of the frame construction processwith all macroblocks in a frame processed in order of increasingmacroblock addresses. For each macroblock, vertical edges are filteredfirst, from left to right, and then horizontal edges are filtered fromtop to bottom. The luma deblocking filter process is performed on four16-sample edges and the deblocking filter process for each chromacomponent is performed on two 8-sample edges, for the horizontaldirection and for the vertical direction, as shown in FIG. 39. Samplevalues above and to the left of the current macroblock that may havealready been modified by the deblocking process operation on previousmacroblocks shall be used as input to the deblocking filter process onthe current macroblock and may be further modified during the filteringof the current macroblock. Sample values modified during filtering ofvertical edges can be used as input for the filtering of the horizontaledges for the same macroblock. A deblocking process can be invoked forthe luma and chroma components separately.

In an example of deringing processing, a 2-D filter can be adaptivelyapplied to smooth out areas near edges. Edge pixels undergo little or nofiltering in order to avoid blurring.

GOP Partitioner

Illustrative examples of processing is described below includingbandwidth map generation, shot detection, and adaptive GOP partitioning,than can be included in the GOP partitioner.

Bandwidth Map Generation

Human visual quality V can be a function of both encoding complexity Cand allocated bits B (also referred to as bandwidth). FIG. 29 is a graphillustrating this relationship. It should be noted that the encodingcomplexity metric C considers spatial and temporal frequencies from thehuman vision point of view. For distortions more sensitive to humaneyes, the complexity value is correspondingly higher. It can typicallybe assume that V is monotonically decreasing in C, and monotonicallyincreasing in B.

To achieve constant visual quality, a bandwidth (B_(i)) is assigned tothe i^(th) object (frame or MB) to be encoded that satisfies thecriteria expressed in the two equations immediately below:

$\begin{matrix}{B_{i} = {B\left( {C_{i},V} \right)}} & (42) \\{B = {\sum\limits_{i}B_{i}}} & (43)\end{matrix}$In the two equations immediately above, C_(i) is the encoding complexityof the i^(th) object. B is the total available bandwidth, and V is theachieved visual quality for an object.

Human visual quality is difficult to formulate as an equation.Therefore, the above equation set is not precisely defined. However, ifit is assumed that the 3-D model is continuous in all variables,bandwidth ratio

$\left( \frac{B_{i}}{B} \right)$can be treated as unchanged within the neighborhood of a (C, V) pair.The bandwidth ratio βi is defined in the equation shown below:

$\begin{matrix}{\beta_{i} = \frac{B_{i}}{B}} & (44)\end{matrix}$

Bit allocation can then be defined as expressed in the followingequations:

$\begin{matrix}{{\beta_{i} = {\beta\left( C_{i} \right)}}{1 = {\sum\limits_{i}\beta_{i}}}{{{for}\left( {C_{i},V} \right)} \in {\delta\left( {C_{0},V_{0}} \right)}}} & (45)\end{matrix}$where δ indicates the “neighborhood.”

The encoding complexity is affected by human visual sensitivity, bothspatial and temporal. Girod's human vision model is an example of amodel that can be used to define the spatial complexity. This modelconsiders fee local spatial frequency and ambient lighting. Theresulting metric is called D_(csat). At a pre-processing point in theprocess, whether a picture is to be intra-coded or inter-coded is notknown and bandwidth ratios for both are generated. Bits are allocatedaccording to the ratio between β_(INTRA) of different video objects. Forintra-coded pictures, the bandwidth ratio is expressed in the followingequation:β_(INTRA)=β_(0INTRA) log₁₀(1+α_(INTRA) Y ² D _(csat))  (46)

In the equation above, Y is the average luminance component of amacroblock, α_(INTRA) is a weighing factor for the luminance square and;D_(csat) term following it, β_(0INTRA) is a normalization factor toguarantee

$1 = {\sum\limits_{i}{\beta_{i}.}}$For example, a value for α_(INTRA)=4 achieves good visual quality.Content information (e.g., a content classification) can be used to setα_(INTRA) to a value that corresponds to a desired good visual qualitylevel for the particular content of the video. In one example, if thevideo content comprises a “talking head” news broadcast, the visualquality level may be set lower because the information image ordisplayable portion of the video may be deemed of less importance thanthe audio portion, and less bits can be allocated to encode the data. Inanother example, if the video content comprises a sporting event,content information may be used to set α_(INTRA) to a value thatcorresponds to a higher visual quality level because the displayedimages may be more important to a viewer, and accordingly more bits canbe allocated to encode the data.

To understand this relationship, it should be noted that bandwidth isallocated logarithmically with encoding complexity. The luminancesquared term Y² reflects the fact that coefficients with largermagnitude use more bits to encode. To prevent the logarithm from gettingnegative values, unity is added to the term in the parenthesis.Logarithms, with other bases can also be used.

The temporal complexity is determined by a measure of a frame differencemetric, which measures the difference between two consecutive framestaking into account the amount of motion (e.g., motion vectors) alongwith a frame difference metric such as the sum of the absolutedifferences (SAD).

Bit allocation for inter-coded pictures can consider spatial as well astemporal complexity. This is expressed below:β_(INTER)=β_(0INTER) log₁₀(1+α_(INTER)·SSD·D _(csat)exp(−γ∥MV_(P)+MV_(N)∥²))  (47)

In the above equation, MV_(P) and MV_(N) are the forward and thebackward motion vectors for the current MB. It can be noted that Y² inthe intra-coded bandwidth formula is replaced by sum of squareddifferences (SSD). To understand the role of ∥MV_(P)+MV_(N)∥² in theabove equation, note the next characteristics of human visual system:areas undergoing smooth, predictable motion (small ∥MV_(P)+MV_(N)∥²)attract attention and can foe tracked by the eye and typically cannottolerate any more distortion than stationary regions. However, areasundergoing fast or unpredictable motion (large ∥MV_(P)+MV_(N)∥²) cannotbe tracked and can tolerate significant quantization. Experiments showthat α_(INTER)=1, γ=0.001 achieves good visual quality.

Shot Detection

An illustrative example of shot detection is described below. Suchcomponents and process can be included m the GOP partitioner 412 (FIG.4).

The motion compensator 23 can be configured to determine bi-directionalmotion information about frames in the video. The motion compensator 23can also be configured to determine one or more difference metrics, forexample, the sum of absolute differences (SAD) or the sum of absolutedifferences (SSD), and calculate other information including luminanceinformation for one or more frames (e.g., macroblock. (MB) luminanceaverages or differences), a luminance histogram difference, and a framedifference metric, examples of which are described in reference toEquations 1-3. The shot classifier can be configured to classify framesin the video into two or more categories of “shots” using informationdetermined by the motion compensator. The encoder is configured toadaptively encode the plurality of frames based on the shotclassifications. The motion compensator, shot classifier, and encoderare described below in reference to Equations 1-10.

FIG. 28 is a block diagram of a preprocessor 202 comprising a processor2831 configured for shot detection and other preprocessing operationsaccording to some aspects. A digital video source can be provided by asource external to the preprocessor 202 as shown in FIG. 4 andcommunicated to a communications module 2836 in the preprocessor 202.The preprocessor 202 contains a storage medium 2825 which communicateswith the processor 2831, both of which communicate with thecommunications module 2836. The processor 2831 includes a motioncompensator 2032, a shot classifier 2833, and other modules forpreprocessing 2034, which can operate to generate motion information,classify shots in frames of the video data, and perform otherpreprocessing tests as described herein. The motion compensator, shotclassier, and other modules can contain processes similar tocorresponding modules in FIG. 4, and can process video to determineinformation described below. In particular, the processor 2831 can havea configuration to obtain metrics indicative of a difference betweenadjacent frames of a plurality of video frames, the metrics comprisingbi-directional motion information and luminance information, determineshot changes in the plurality of video frames based on said metrics, andadaptively encode the plurality of frames based on the shot changes. Insome aspects, the metrics can be calculated by a device or processexternal to the processor 2831, which can also be external to thepreprocessor 202, and communicated to the processor 2831, eitherdirectly or indirectly via another device or memory. The metrics canalso be calculated by the processor 2831, for example, by the motioncompensator 2832.

The preprocessor 202 provides video and metadata for further processing,encoding, and transmission to other devices, for example, terminals 6(FIG. 1). The encoded video can be, in some aspects, scalablemulti-layered encoded video which can comprise a base layer and anenhancement layer. Scalable layer encoding is further described inco-pending U.S. patent application Ser. No. 11/373,604 entitled“Scalable Video Coding With Two Layer Encoding And Single LayerDecoding” owned by the assignee hereof, and which is incorporated byreference in its entirety herein.

The various illustrative logical blocks, components, modules, andcircuits described in connection with FIG. 28, and other examples andfigures disclosed herein may be implemented or performed, in someaspects, with a general purpose processor, a digital signal processor(DSP), an application specific integrated circuit (ASIC), a fieldprogrammable gate array (FPGA) or other programmable logic device,discrete gate or transistor logic, discrete hardware components, or anycombination thereof designed to perform the functions described herein.A general purpose processor such as the one shown in FIG. 28 may be amicroprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

Video encoding usually operates on a structured group of pictures (GOP).A GOP normally starts with an intra-coded frame (I-frame), followed by aseries of P (predictive) or B (bi-directional) frames. Typically, anI-frame can store all the data to display the frame, a B-frame relies ondata in the preceding and following frames (e.g., only containing datachanged from the preceding frame or is different from data in the nextframe), and a P-frame contains data that has changed from the precedingframe.

In common usage, I-frames are interspersed with P-frames and B-frames inencoded video. In terms of size (e.g., number of bits used to encode theframe), I-frames are typically much larger than P-frames, which in turnare larger than B-frames. For efficient encoding, transmission anddecoding processing, the length of a GOP should be long enough to reducethe efficient loss from big I-frames, and short enough to fight mismatchbetween encoder and decoder, or channel impairment. In addition, macroblocks (MB) in P frames can be intra coded for the same reason.

Scene change detection can be used for a video encoder to determine aproper GOP length and insert I-frames based on the GOP length, insteadof inserting an I-frame at a fixed interval. In a practical streamingvideo system, the communication channel is usually impaired by biterrors or packet losses. Where to place I frames or I MBs maysignificantly impact decoded video quality and viewing experience. Oneencoding scheme is to use intra-coded frames for pictures or portions ofpictures that have significant change from collocated previous picturesor picture portions. Normally these regions cannot be predictedeffectively and efficiently with motion estimation, and encoding can bedone more efficiently if such regions are exempted from inter-framecoding techniques (e.g., encoding using B-frames and P-frames). In thecontext of channel impairment, those regions are likely to suffer fromerror propagation, which can be reduced or eliminated (or nearly so) byintra-frame encoding.

Portions of the GOP video can be classified into two or more categories,where each region can have different intra-frame encoding criteria thatmay depend on the particular implementation. As an example, the videocan be classified into three categories: abrupt scene changes,cross-fading and other slow scene changes, and camera flashlights.Abrupt scene changes includes frames that are significantly differentfrom the previous frame, usually caused by a camera operation. Since thecontent of these frames is different from that of the previous frame,the abrupt scene change frames should be encoded as I frames.Cross-fading and other slow scene changes includes slow switching ofscenes, usually caused by computer processing of camera shots. Gradualblending of two different scenes may look more pleasing to human eyes,but poses a challenge to video coding. Motion compensation cannot reducethe bitrate of those frames effectively, and more intra MBs can beupdated for these frames.

Camera flashlights, or camera flash events, occur when the content of aframe includes camera flashes. Such flashes are relatively short induration (e.g., one frame) and extremely bright such that the pixels ina frame portraying the flashes exhibit unusually high luminance relativeto a corresponding area on an adjacent frame. Camera flashlights shiftthe luminance of a picture suddenly and swiftly. Usually the duration ofa camera flashlight is shorter than the temporal masking duration of thehuman vision system (HVS), which is typically defined to be 44 ms. Humaneyes are not sensitive to the quality of these short bursts ofbrightness and therefore they can be encoded coarsely. Because theflashlight frames cannot be handled effectively with motion compensationand they are bad prediction candidate for future frames, coarse encodingof these frames does not reduce the encoding efficiency of futureframes. Scenes classified as flashlights should not be used to predictother frames because of the “artificial” high luminance, and otherframes cannot effectively be used to predict these frames for the samereason. Once identified, these frames can be taken out because they canrequire a relatively high amount of processing. One option is to removethe camera flashlight frames and encode a DC coefficient in their place;such a solution is simple, computationally fast and saves many bits.

When any of the above frames are detected, a shot event is declared.Shot detection is not only useful to improve encoding quality, it canalso aid in identifying video content searching and indexing. One aspectof a scene detection process is described hereinbelow.

FIG. 30 illustrates a process 3000 that operates on a GOP and can beused in some aspects to encode video based on shot detection in videoframes, where portions of the process 3000 (or sub-processes) aredescribed and illustrated with reference to FIGS. 30-40. The processor2831 can be configured to incorporate process 3000. After process 3000starts, it proceeds to block 3042 where metrics (information) areobtained for the video frames, the metrics including informationindicative of a difference between adjacent frames. The metrics includesbidirectional motion information and luminance-based information that issubsequently to determine changes that occurred between adjacent frameswhich can be used for shot classification. Such metrics can be obtainedfrom another device or process, or calculated by, for example, processor2831. Illustrative examples of metrics generation are described inreference to process A in FIG. 31.

Process 3000 then proceeds to block 3044 where shot changes in the videoare determined based on the metrics. A video frame can be classifiedinto two or more categories of what type of shot is contained in theframe, for example, an abrupt scene change, a slowly changing scene, ora scene containing high luminance values (camera flashes). Certainimplementations encoding may necessitate other categories. Anillustrative example of shot classification is described in reference toprocess B in FIG. 32, and in more detail with reference to processes D,E, and F in FIGS. 34-36, respectively.

Once a frame is classified, process 3000 proceeds to block 3046 wherethe frame can be encoded, or designated for encoding, using the shotclassification results. Such results can influence whether to encode theframe with an intra-coded frame or a predictive frame (e.g., P-frame orB-frame). Process C in FIG. 33 shows an example of an encoding schemeusing the shot results.

FIG. 31 illustrates an example of a process for obtaining metrics of thevideo. FIG. 31 illustrates certain steps that occur in block 3042 ofFIG. 30. Referring still to FIG. 31, in block 3152, process A obtains ordetermines bi-directional motion estimation and compensation informationof the video. The motion compensator 2832 of FIG. 28 can be configuredto perform bi-directional motion estimation on the frames and determinemotion compensation information that can be used for subsequent shotclassification. Process A then proceeds to block 3154 where it generatesluminance information including a luminance difference histogram for acurrent or selected frame and one or more adjacent frames. Lastly,process A then continues to block 3156 where a metric is calculated thatindicative of the shot contained in the frame. One such metric is aframe difference metric which is shown in two examples in Equations 4and 10. Illustrative examples of determining motion information,luminance information, and a frame difference metric are describedbelow.

Motion Compensation

To perform bi-directional motion estimation/compensation, a videosequence can be preprocessed with a bidirectional motion compensatorthat matches every 8×8 block of the current frame with blocks in two ofthe frames most adjacent neighboring frames, one in the past, and one inthe future. The motion compensator produces motion vectors anddifference metrics for every block. FIG. 37 illustrates this concept,showing ah example of matching pixels of a current frame C to a pastframe P and a future (or next) frame N, and depicts motion vectors tothe matched pixels (past motion vector MV_(P) and future motion vectorMV_(N). A brief description of an illustrative aspect of bi-directionalmotion vector generation and related encoding follows below.

FIG. 40 illustrates an example of a motion vector determination processand predictive frame encoding in, for example, MPEG-4. The processdescribed in FIG. 40 is a more detailed illustration of an exampleprocess that can take place in block 3152 of FIG. 31. In FIG. 40,current picture 4034 is made up of 5×5 macroblocks, where the number ofmacroblocks in this example is arbitrary. A macroblock is made up of16×16 pixels. Pixels can be defined by an 8-bit luminance value (Y) andtwo 8-bit chrominance values (Cr and Cb).

In MPEG, Y, Cr and Cb components can be stored in a 4:2:0 format, wherethe Cr and Cb components are down-sampled by 2 in the X and the Ydirections. Hence, each macroblock would consist of 256 Y components, 64Cr components and 64 Cb components. Macroblock 4036 of current picture4034 is predicted from reference picture 4032 at a different time pointthan current picture 4034. A search is made in reference picture 4032 tolocate best matching macroblock 4038 that is closest, in terms of Y, Crand Cb values to current macroblock 4036 being encoded. The location ofbest matching macroblock 138 in reference picture 4032 is encoded inmotion vector 4040. Reference picture 4032 can be an I-frame or P-framethat a decoder will have reconstructed prior to the construction ofcurrent picture 4034. Best matching macroblock 4038 is subtracted fromcurrent macroblock 40 (a difference for each of the Y, Cr and Cbcomponents is calculated) resulting in residual error 4042. Residualerror 4042 is encoded with 2D Discrete Cosine Transform (DCT) 4044 andthen quantized 4046. Quantization 4046 can be performed to providespatial compression by, for example, allotting fewer bits to the highfrequency coefficients while allotting more bits to the low frequencycoefficients. The quantized coefficients of residual error 4042, alongwith motion vector 4040 and reference picture 4034 identifyinginformation, are encoded information representing current macroblock4036. The encoded information can be stored in memory for future use oroperated on for purposes of, for example, error correction or imageenhancement, or transmitted over network 140.

The encoded quantized coefficients of residual error 4042, along withencoded motion vector 4040 can be used to reconstruct current macroblock4036 in the encoder for use as part of a reference frame for subsequentmotion estimation and compensation. The encoder can emulate theprocedures of a decoder for this P-frame reconstruction. The emulationof the decoder will result in both the encoder and decoder working withthe same reference picture. The reconstruction process, whether done inan encoder, for further inter-coding, or in a decoder, is presentedhere. Reconstruction of a P-frame can be started after the referenceframe (or a portion of a picture or frame that is being referenced) isreconstructed. The encoded quantized coefficients are dequantized 4050and then 2D Inverse DCT, or IDCT, 4052 is performed resulting in decodedor reconstructed residual error 4054. Encoded motion vector 4040 isdecoded and used to locate the already reconstructed best matchingmacroblock 4056 in die already reconstructed reference picture 4032.Reconstructed residual error 4054 is then added to reconstructed bestmatching macroblock 4056 to form reconstructed macroblock 4058.Reconstructed macroblock 4058 can be stored in memory, displayedindependently or in a picture with other reconstructed macroblocks, orprocessed further for image enhancement.

Encoding using B-frames (or any section coded wife bi-directionalprediction) can exploit temporal redundancy between a region in acurrent picture and a best matching prediction region in a previouspicture and a best matching prediction region in a subsequent picture.The subsequent best matching prediction region and the previous bestmatching prediction region are combined to form a combinedbi-directional predicted region. The difference between the currentpicture region and the best matching combined bi-directional predictionregion is a residual error (or prediction error). The locations of thebest matching prediction region in the subsequent reference picture andthe best matching prediction region in the previous reference picturecan be encoded in two motion vectors.

Luminance Histogram Difference

The motion compensator can produce a difference metric for every block.The difference metric can be a sum of square difference (SSD) or a sumof absolute difference (SAD). Without loss of generality, here SAD isused as an example.

For every frame, a SAD ratio is calculated as below:

$\begin{matrix}{\gamma = \frac{ɛ + {SAD}_{P}}{ɛ + {SAD}_{N}}} & (48)\end{matrix}$where SAD_(P) and SAD_(N) are the sum of absolute differences of theforward and the backward difference metric, respectively. It should benoted that the denominator contains a small positive number e to preventthe “divide-by-zero” error. The nominator also contains an ε to balancethe effect of the unity in the denominator. For example, if the previousframe, the current frame, and the next frame are identical, motionsearch should yield SAD_(P)=SAD_(N)=0. In this case, the abovecalculation generators γ=1 instead of 0 or infinity.

A luminance histogram can be calculated for every frame. Typically themultimedia images have a luminance depth (e.g., number of “bins”) ofeight bits. The luminance depth used for calculating the luminancehistogram according to some aspects can be set to 16 to obtain thehistogram. In other aspects, the luminance depth can be set to anappropriate number which may depend upon the type of data beingprocessed, the computational power available, or other predeterminedcriteria. In some aspects, the luminance depth can be set dynamicallybased on a calculated or received metric, such as the content of thedata.

Equation 49 illustrates one example of calculating a luminance histogramdifference (lambda):

$\begin{matrix}{\lambda = \frac{\sum\limits_{i = 1}^{16}{{N_{Pi} - N_{Ci}}}}{N}} & (49)\end{matrix}$where N_(Pi) is the number of blocks in the i^(th) bin for the previousframe, and N_(Ci) is the number of blocks in the i^(th) bin for thecurrent frame, and N is the total number of blocks in a frame. If theluminance histogram difference of the previous and the current frame arecompletely dissimilar (or disjoint), then λ=2.

A frame difference metric D, discussed in reference to block 56 of FIG.5, can be calculated, as shown in Equation 50:

$\begin{matrix}{D = {\frac{\gamma_{C}}{\gamma_{P}} + {A\;{\lambda\left( {{2\lambda} + 1} \right)}}}} & (50)\end{matrix}$where A is a constant chosen by application, and

${\gamma_{C} = \frac{ɛ + {SAD}_{P}}{ɛ + {SAD}_{N}}},$and

$\gamma_{P} = {\frac{ɛ + {SAD}_{PP}}{ɛ + {SAD}_{C}}.}$

FIG. 32 illustrates an example of a process B which determines threecategories of shot (or scene) changes using metrics obtained ordetermined for the video. FIG. 32 illustrates certain steps occurring inone aspect of block 3044 of FIG. 30. Again referring to FIG. 32, inblock 3262, process B first determines if the frame meets criteria to bedesignated an abrupt scene change. Process D in FIG. 34 illustrates anexample of this determination. Process B then proceeds to block 3264where it determines of the frame is part of a slowly changing scene.Process C in FIG. 35 illustrates an example of determining a slowchanging scene. Finally, at block 3366 process B determines if the framecontains camera flashes, in other words, large luminance valuesdiffering from the previous frame. Process F in FIG. 36 illustrates anexample of determining a frame containing camera flashes. Anillustrative example of these processes are described below.

Abrupt Scene Change

FIG. 34 is a flow diagram illustrating a process of determining abruptscene changes. FIG. 34 further elaborates certain steps that can occurin some aspects of block 3262 of FIG. 32. At block 3482 checks if theframe difference metric D meets the criterion shown in Equation 51:

$\begin{matrix}{D = {{\frac{\gamma_{C}}{\gamma_{P}} + {A\;{\lambda\left( {{2\lambda} + 1} \right)}}} \geq T_{1}}} & (51)\end{matrix}$where A is a constant chosen by application, and T₁ is a threshold. Ifthe criteria is met, at block 3484 process D designates the frame as anabrupt scene change and, in this example, no further shot classificationis necessary.

In one example simulation shows, setting A=1, and T₁=5 achieve gooddetection performance. If the current frame is an abrupt scene changeframe, then γ_(C) should be large and γ_(P) should be small. The ratio

$\frac{\gamma_{C}}{\gamma_{P}}$can be used instead of γ_(C) alone so that the metric is normalized tothe activity level of the context.

It should be noted that the above criterion uses the luminance histogramdifference lambda (λ) in a non-linear way. FIG. 39 illustrates λ*(2λ+1)is a convex function. When λ is small (e.g., close to zero), it isbarely preemphasis. The larger λ becomes, the more emphasis is conductedby the function. With this pre-emphasis, for any λ larger than 1.4, anabrupt scene change is detected if the threshold T₁ is set to be 5.

Cross-Fading and Slow Scene Changes

FIG. 35 further illustrates further details of some aspects that canoccur in block 3264 of FIG. 32. Referring to FIG. 35, at block 3592process E determines if the frame is part of a series of framesdepicting a slow scene change. Process E determines that the currentframe is a cross-fading or other slow scene change if the framedifference metric D is less than the first threshold value T₁ andgreater or equal to a second threshold value T₂ as illustrated inEquation 52:T ₂ ≦D<T ₁  (52)for a certain number of continuous frames, where T₁ is the samethreshold used above and T₂ is another threshold value. Typically, theexact value of T₁ and T₂ are determined by normal experimentationbecause of the difference in implementations that are possible. If thecriteria is met, at block 94 process E classifies the frame as part of aslow changing scene shot classification for the selected frame ends.Camera Flashlight Events

Process F shown in FIG. 36 is an example of a process that can determineif the current frame comprises camera flashlights. In this illustrativeaspect camera, the luminance histogram statistics are used to determineif the current frame comprises camera flashlights. Process F determinescamera flash events are in the selected frame by first determining ifthe luminance of a current frame is greater than the luminance of theprevious frame and the luminance of the next frame, shown at block 3602.If not, the frame is not a camera flash event; but if so it may be. Atblock 3604, Process F determines whether the backwards difference metricis greater than a threshold T₃, and if the forwards difference metric isgreater than a threshold T₄; if both these conditions are satisfied, atblock 3606 process F classifies the current frame as having cameraflashlights. In one example, at block 3602, process F determines if theaverage luminance of the current frame minus the average luminance ofthe previous frame is equal of exceeds a threshold T₃, and process Fdetermines if the average luminance of minus the average luminance ofthe next frame is greater than or equal to the threshold T₃, as shown inEquations 53 and 54:Y _(C) − Y _(P) ≧T ₃  (53)Y _(C) − Y _(N) ≧T ₃  (54)

If the criterion is not met, the current frame is not classified ascomprising camera flashlights and process F returns. If the criterion ismet, process F proceeds to block 3604 where it determines if a backwardsdifference metric SAD_(P) and a forward difference metric SAD_(N) aregreater than a certain threshold T4, as illustrated in Equations 55 and56 below:SAD_(P) ≧T ₄  (55)SAD_(N) ≧T ₄  (56)where Y _(C) is the average luminance of the current frame, Y _(P) isthe average luminance of the previous frame, Y _(N) is the averageluminance of the next frame, and SAD_(P) and SAD_(N) are the forward andbackward difference metrics associated with the current frame. If thecriterion is not met, process F returns.

Values of T₃ are typically determined by normal experimentation as theimplementation of the described processes can result in differences inoperating parameters including threshold values. SAD values are includedin the determination because camera flashes typically take only oneframe, and due to the luminance, difference, this frame cannot bepredicted welt using motion compensation from both the forward and thebackward direction.

In some aspects, one or more of the threshold values T₁, T₂, T₃, and T₄are predetermined and such values are incorporated into the shotclassifier in the encoding device. Typically, these threshold values areselected through testing of a particular implementation of shotdetection. In some aspects, one or more of the threshold values T₁, T₂,T₃, and T₄ can be set during processing (e.g., dynamically) based onusing information (e.g., metadata) supplied to die shot classifier orbased on information calculated by the shot classifier itself.

Referring now to FIG. 33 which shows a process C for determiningencoding parameters for the video, or for encoding the video, based onthe shot classification of the selected frame. At block 3370 process Cdetermines if the selected frame was classified as an abrupt scenechange. If so, at block 3371 the current frame is classified as anabrupt scene change, and the frame can be encoded as an I-frame and aGOP boundary can be determined. If not, process C proceeds to block3372; if the current frame is classified as a portion of a slowlychanging scene, at block 3373 the current frame, and other frames in theslow changing scene can be encoded as a predictive frame (e.g., P-frameor B-frame). Process C then proceeds to block 3374 where it checks ifthe current frame was classified as a flashlight scene comprising cameraflashes. If so, at block 3375 the frame can be identified for specialprocessing, for example, removal, replication of a previous frame orencoding a particular coefficient for the frame. If not, noclassification of the current frame was made and the selected frame canbe encoded in accordance with other criteria, encoded as an I-frame, ordropped. Process C can be implemented in an encoder.

In the above-described aspect, the amount of difference between theframe to be compressed and its adjacent two frames is indicated by aframe difference metric D. If a significant amount of a one-wayluminance change is detected, it signifies a cross-fade effect in theframe. The more prominent the cross-fade is, the more gain may beachieved by using B frames. In some aspects, a modified frame differencemetric is used as shown in Equation 57 below:

$\begin{matrix}{D_{1} = \left\{ \begin{matrix}{{\left( {1 - \alpha + {2\alpha\frac{{\mathbb{d}_{P}{- \mathbb{d}_{N}}}}{\mathbb{d}_{P}{+ \mathbb{d}_{N}}}}} \right) \times D},} & \begin{matrix}{{{{if}\mspace{14mu} Y_{P}} - \Delta} \geq Y_{C} \geq {Y_{N} + {\Delta\mspace{14mu}{or}}}} \\{{{Y_{P} + \Delta} \leq Y_{C} \leq {Y_{N} - \Delta}},}\end{matrix} \\{D,} & {{otherwise},}\end{matrix} \right.} & (57)\end{matrix}$where d_(P)=|Y_(C)−Y_(P)| and d_(N)=|Y_(C)−Y_(N)| are the lumadifference between the current frame and the previous frame, and theluma difference between the current frame and the next frame,respectively, Δ represents a constant feat can be determined in normalexperimentation as it can depend on the implementation, and α is aweighting variable having a value between 0 and 1.

The modified frame difference metric D₁ is only different from theoriginal frame difference metric D if a consistent trend of luma shiftis observed and the shift strength is large enough. D₁ is equal to orless than D. If the change of luma is steady (d_(P)−d_(N)), the modifiedframe difference metric D₁ is lower than the original frame differencemetric D with the lowest ratio of (1−α).

Table 1 below shows performance improvement by adding abrupt scenechange detection. The total number of I-frames in both thenon-scene-change (NSC) and the scene-change (SC) Cases are approximatelythe same. In the NSC case, I-frames are distributed uniformly among thewhole sequence, while in the SC case, I-frames are only assigned toabrupt scene change frames.

It can be seen that typically 0.2-0.3 dB improvement can be achievePSNR-wise. Simulation results show that the shot detector is veryaccurate in determining the shot events above-mentioned. Simulation offive clips with normal cross-fade effect shows that at Δ=5.5 and α=0.4,a PSNR gain of 0.226031 dB is achieved at the same bitrate.

TABLE 1 Simulation Results Of Abrupt Scene Change Detection MetricSequence Bitrate (kbps) Avg. QP PSNR (dB) Animation NSC 226.2403 31.169635.6426 Animation SC 232.8023 29.8171 36.4513 Music NSC 246.6394 32.852435.9337 Music SC 250.0994 32.3209 36.1202 Headline NSC 216.9493 29.830438.9804 Headline News SC 220.2512 28.9011 39.3151 Basketball NSC256.8726 33.1429 33.5262 Basketball SC 254.9242 32.4341 33.8635Adaptive GOP Structure

An illustrative example of adaptive GOP structure operations aredescribed below. Such operations can be included in the GOP partitioner412 of FIG. 412. MPEG2, an older video compression standard, does notrequire that the GOP have a regular structure, though one can beimposed. The MPEG2 sequence always begins with ah I frame, i.e., onewhich has been encoded without reference to previous pictures. The MPEG2GOP format is usually prearranged at the encoder by fixing the spacingin the GOP of the P or predictive pictures that follow the I frame. Pframes are pictures that have been in part predicted from previous I orP pictures. The frames between the starting I frame, and the succeedingP frames are encoded as B frames, A “B” frame (B stands forbi-directional) can use the previous and next I or P pictures eitherindividually or simultaneously as reference. The number of bits used toencode an I-frame on the average exceeds the number of bits used toencode a P-frame; likewise the number of bits used to encode a P-frameon the average exceeds that of a B-frame. A skipped frame, if it isused, may use no bits for its representation.

One benefit of using P-frames and B-frames, and in more recentcompression algorithms, the skipping of frames is that it is possible toreduce video transmission sizes. When temporal redundancy is high—e.g.,when there is little change from picture to picture—use of P, B, orskipped pictures efficiently represents the video stream, because I or Ppictures decoded earlier are used later as references to decode other Por B pictures.

A group of pictures partitioner adaptively encodes frames to minimizetemporal redundancy. Differences between frames are quantified and adecision to represent the picture by a I, P, B, or skipped frame isautomatically made after suitable tests are performed on the quantifieddifferences. The processing in a GOP partitioner and is aided by otheroperations of die preprocessor 202, which provides filtering for noiseremoval.

Adaptive encoding process has advantages not available in a “fixed”encoding process. A fixed process ignores the possibility that littlechange in content has taken place; however, an adaptive procedure allowsfar more B frames to be inserted between each I and P, or two P frames,thereby reducing the number of bits used to adequately represent thesequence of frames. Conversely, e.g., in a fixed encoding process, whenthe change in video content is significant, the efficiency of P framesis greatly reduced because the difference between the predicted and thereference frames is too large. Under these conditions, matching objectsmay fail out of the motion search regions, or the similarity betweenmatching objects is reduced due to distortion caused by changes incamera angle. An adaptive encoding process may beneficially be used toOptionally determine when P frames should be encoded.

In the system disclosed herein, the types of conditions described aboveare automatically sensed. The adaptive encoding process described hereinis flexible and is made to adapt to these changes in content. Theadaptive encoding process evaluates a frame difference metric, which canbe thought of as measure of distance between frames, with the sameadditive properties of distance. In concept, given frames F₁, F₂, and F₃having the inter-frame distances d₁₂ and d₂₃, the distance between F₁and F₃ is taken as being at least d₁₂+d₂₃. Frame assignments are made onthe basis of this distance-like metric and other measures.

The GOP partitioner 412 operates by assigning picture types to frames asthey are received. The picture type indicates the method of predictionthat may be used to code each block:

I-pictures are coded without reference to other pictures. Since theystand alone they provide access points in the data stream where decodingcan begin. An I encoding type is assigned to a frame if the “distance”to its predecessor frame exceeds a scene change threshold.

P-pictures can use the previous I or P pictures for motion compensatedprediction. They use blocks in the previous fields or frames that may bedisplaced from the block being predicted as a basis for encoding. Afterthe reference block is subtracted from the block being considered, theresidual block is encoded, typically using the discrete cosine transformfor the elimination of spatial redundancy. A P encoding types isassigned to a frame if the “distance” between it and the last frameassigned to be a P frame exceeds a second threshold, which is typicallyless than the first.

B-frame pictures can use the previous and next P- or I-pictures formotion compensation as described above. A block in a B picture can beforward, backward or bi-directionally predicted; or it could beintra-coded without reference to other frames. In H.264 a referenceblock can be a linear combination of as many as 32 blocks from as manyframes. If the frame cannot be assigned to be an I or P type, it isassigned to be a B type, if the “distance” from it to its immediatepredecessor is greater than a third threshold, which typically is lessthan the second threshold. If the frame cannot be assigned to become aB-frame encoded, it is assigned to “skip frame” status. This frame canbe skipped because it is virtually a copy of a previous frame.

Evaluating a metric that quantifies the difference between adjacentframes in the display order is the first part of this processing thattakes place in GOP partitioner 412. This metric is the distance referredto above; with it, every frame is evaluated for its proper type. Thus,the spacing between the I and adjacent P, or two successive P frames,can be variable. Computing the metric begins by processing the videoframes with a block-based motion compensator, a block being the basicunit of video compression, composed usually of 16×16 pixels, thoughother block sizes such as 8×8, 4×4 and 8×16 are possible. For framesconsisting of two deinterlaced fields that are present at the output,the motion compensation is done on a field basis, the search for thereference blocks taking place in fields rather than frames. For a blockin the first field of the current frame a forward reference block isfound in fields of the frame that follows it; likewise a backwardreference block found in fields of the frame that immediately precedesthe current field. The current blocks are assembled into a compensatedfield. The process continues with the second field of the frame. The twocompensated fields are combined to form a forward and a backwardcompensated frame.

For frames created in the inverse telecine 406, the search for referenceblocks may be on a frame basis only, since only reconstructed filmframes are generated. Two reference blocks and two differences, forwardand backward, are found, leading also to a forward and backwardcompensated frame. In summary, the motion compensator produces motionvectors and difference metrics for every block. Note that thedifferences in the metric are evaluated between a block in the field orframe being considered and a block that best matches it, either in apreceding field or frame or a field or frame that immediately followsit, depending on whether a forward or backward difference is beingevaluated. Only luminance values enter into this calculation.

The motion compensation step thus generates two sets of differences.These are between blocks of current values of luminance and theluminance values in reference blocks taken from frames that areimmediately ahead and immediately behind the current one in time. Theabsolute value of each forward and each backward difference isdetermined for each pixel in a block and each is separately summed overthe entire frame. Both fields are included in the two summations whenthe deinterlaced NTSC fields that comprise a frame are processed. Inthis way, SAD_(P), and SAD_(N), the summed absolute values of theforward and backward differences are found.

For every frame a SAD ratio is calculated using the relationship,

$\begin{matrix}{\gamma = \frac{ɛ + {SAD}_{P}}{ɛ + {SAD}_{N}}} & (58)\end{matrix}$where SAD_(P) and SAD_(N) are the summed absolute values of the forwardand backward differences respectively. A small positive number is addedto the numerator ε to prevent the “divide-by-zero” error. A similar εterm is added to the denominator, further reducing the sensitivity of γwhen either SAD_(P) or SAD_(N) is close to zero.

In an alternate aspect, the difference can be the SSD, the sum ofsquared differences, and SAD, the sum of absolute differences, or theSATD, in which the blocks of pixel values are transformed by applyingthe two dimensional Discrete Cosine Transform to them before differencesin block elements are taken. The sums are evaluated over the area ofactive video, though a smaller area may be used in other aspects.

The luminance histogram of every frame as received (non-motioncompensated) is also computed. The histogram operates on the DCcoefficient, i.e., the (0,0) coefficient, in the 16×16 array ofcoefficients that is the result of applying the two dimensional DiscreteCosine Transform to the block of luminance values if it were available.Equivalently the average value of the 256 values of luminance in the16×16 block may be used in the histogram. For images whose luminancedepth is eight bits, the number of bins is set at 16. The next metricevaluates the histogram difference

$\begin{matrix}{\lambda = {\frac{1}{N}{\sum\limits_{i = 1}^{16}\;{{N_{Pi} - N_{Ci}}}}}} & (59)\end{matrix}$

In the above, N_(Pi) is the number of blocks from the previous frame inthe i^(th) bin, and N_(ci) is the number of blocks from the currentframe that belong in the i^(th) bin, N is the total number of blocks ina frame.

These intermediate results are assembled to form the current framedifference metric as

$\begin{matrix}{{M = {\frac{\gamma_{C}}{\gamma_{P}} + {\lambda\left( {{2\lambda} + 1} \right)}}},} & (60)\end{matrix}$

where γ_(C) is the SAD ratio based on the current frame and γ_(P) is theSAD ratio based on the previous frame. If a scene has smooth motion andits luma histogram barely change, then M≈1. If the current framedisplays an abrupt scene change, then γ_(C) will be large and γ_(P)should be small. The ratio

$\frac{\gamma_{C}}{\gamma_{P}}$instead of γ_(C) alone is used so that the metric is normalized to theactivity level of the contest.

Dataflow 4100 in FIG. 40 illustrates certain components that may be usedto compute the frame difference metric. Preprocessor 4125 deliversinterlaced fields in the case of video having a NTSC source, and framesof film images when the source of the video is the result of inversetelecine to the bi-directional motion compensator 4133. Thebi-directional motion compensator 4133 operates on a field (or frame inthe case of a cinematic source of video) by breaking it into blocks of16×16 pixels and comparing each block to all 16×16 blocks in a definedarea of a field of the previous frame. The block which provides the bestmatch is selected and subtracted from the current block. The absolutevalues of the differences is taken and the result summed over the 256pixels that comprise the current block. When this is done for allcurrent blocks of the field, and then for both fields the quantitySAD_(N), the backward difference metric has been computed by a backwarddifference module 4137. A similar procedure may be performed by aforward difference module 4136. The forward difference module 4136 usesthe frame which is immediately ahead of the current one in time as asource of reference blocks to develop the SAD_(P), the forwarddifference metric. The same estimation process, albeit done using therecovered film frames, takes place when the input frames are formed inthe inverse telecine. The histograms that can be used to complete thecomputation of the frame difference metric may be formed in histogramdifference module 4141. Each 16×16 block is assigned to a bin based onthe average value of its luminance. This information is formed by addingall 256 pixel luminance values in a block together, normalizing it by256 if desired, and incrementing the count of the bin into which theaverage value would have been placed. The calculation is done once foreach pre-motion compensated frame, the histogram for the current framebecoming the histogram for the previous frame when a new current framearrives. The two histograms are differenced and normalized by the numberof blocks in histogram difference module 4141 to form λ, defined byEquation 59. These results are combined in frame difference combiner4143, which uses the intermediate results found in histogram differencemodule 4139, forward and backward difference modules 4136 and 4136 toevaluate the current frame difference defined in Equation 60.

The system of flowchart 4100 and components or steps thereof, can beimplemented by hardware, software, firmware, middleware, microcode, orany combination thereof. Each functional component of flowchart 4100,including the preprocessor 4135, the bidirectional motion compensator4133, the toward and backward difference metric modules 4136 and 4137,the histogram, difference module 4141, and the frame difference metriccombiner 4143, may be realized as a standalone component, incorporatedas hardware, firmware, middleware in a component of another device, orbe implemented in microcode or software that is executed on theprocessor, or a combination thereof. When implemented in software,firmware, middleware or microcode, the program code or code segmentsthat perform the desired tasks may be stored in a machine readablemedium such as a storage medium. A code segment may represent aprocedure, a function, a subprogram, a program, a routine, a subroutine,a module, a software package, a class, or any combination ofinstructions, data structures, or program statements. A code segment maybe coupled to another code segment or a hardware circuit by passingand/or receiving information, data, arguments, parameters, or memorycontents.

The received and processed data can be stored in a storage medium whichcan include, for example, a chip configured storage medium (e.g., ROM,RAM) or a disc-type storage medium (e.g., magnetic or optical) connectedto a processor. In some aspects, the combiner 4143 can contain part orall of the storage medium. Flowchart 4200 in FIG. 41 illustrates aprocess of assigning compression types to frames. In one aspect M, thecurrent frame difference defined in Equation 3, is the basis for alldecisions made with respect to frame assignments. As decision block 4253indicates, if a frame under consideration is the first in a sequence,the decision path marked YES is followed to block 4255, therebydeclaring the frame to be an I frame. The accumulated frame differencesis set to zero in block 4257, and the process returns (in block 4258) tothe start block 4253. If the frame being considered is not the firstframe in a sequence, the path marked NO is followed from block 4253where the decision was made, and in test block 4259 the current framedifference is tested against the scene change threshold. If the currentframe difference is larger than that threshold, the decision path markedYES is followed to block 4255, again leading to the assignment of anI-frame. If the current frame difference is less than the scene changethreshold, the NO path is followed to block 4261 where the current framedifference is added the accumulated frame difference.

Continuing through the flowchart at decision block 4263, the accumulatedframe difference is compared with threshold t, which is in general lessthan the scene change threshold. If the accumulated frame difference islarger than t, control transfers to block 4265, and the frame isassigned to be a P frame; the accumulated frame difference is then resetto zero in step 4267. If the accumulated frame difference is less thant, control transfers from block 4263 to block 4269. There the currentframe difference is compared with τ, which is less than t. If thecurrent frame difference is smaller than τ, the frame is assigned to beskipped in block 4273; if the current frame difference is larger than τ,the frame is assigned to be a β frame.

In an alternate aspect another frame encoding complexity indicator M* isdefined asM*=M×min(1,α max(0,SAD_(P) −s)×max(0,MV_(P) −m)),  (61)where α is a scaler, SAD_(P) is the SAD with forward motioncompensation, MV_(P) is the sum of lengths measured in pixels of themotion vectors from the forward motion compensation, and s and m are twothreshold numbers that render the frame encoding complexity indicator tozero if SAD_(P) is lower than s or MV_(P) is lower than m. M* would beused in place of the current frame difference in flowchart 4200 of FIG.41. As can be seen, M* is different from M only if the forward motioncompensation shows a low level of movement. In this case, M is smallerthan M.

It is noted that the shot detection and encoding aspects describedherein may be described as a process which is depicted as a flowchart, aflow diagram, a structure diagram, or a block diagram. Although theflowcharts shown in the figures may describe operations as a sequentialprocess, many operations can be performed in parallel or concurrently.In addition, the order of operations may be re-arranged. A process istypically terminated when its operations are completed. A process maycorrespond to a method, a function, a procedure, a subroutine, asubprogram, etc. When a process corresponds to a function, itstermination corresponds to a return of the function to the callingfunction or the main function.

It should also be apparent to those skilled in the art that one or moreelements of a device disclosed herein may be rearranged withoutaffecting the operation of the device. Similarly, one or more elementsof a device disclosed herein may be combined without affecting theoperation of the device. Those of ordinary skill in the art wouldunderstand that information and multimedia data may be represented usingany of a variety of different technologies and techniques. Those ofordinary skill would further appreciate that the various illustrativelogical blocks, modules, and algorithm steps described in connectionwith the examples disclosed herein may be implemented as electronichardware, firmware, computer software, middleware, microcode, orcombinations thereof. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. Whether such functionality is implemented ashardware or software depends upon the particular application and designconstraints imposed on the overall system. Skilled artisans mayimplement the described functionality in varying ways for eachparticular application, but such implementation decisions should not beinterpreted as causing a departure from the scope of the disclosedmethods.

For example, the steps of a method or algorithm described in connectionwith the shot detection and encoding examples and Figures disclosedherein may be embodied directly in hardware, in a software moduleexecuted by a processor, or in a combination of the two. The methods andalgorithms are particularly applicable to communication technologyincluding wireless transmissions of video to cell phones, computers,laptop computers, PDA's and all types of personal and businesscommunication devices, software module may reside in RAM memory, flashmemory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, aremovable disk, a CD-ROM, or any other form of storage medium known inthe art. An exemplary storage medium is coupled to the processor suchthat the processor can read information from, and write information to,the storage medium. In the alternative, the storage medium may beintegral to the processor. The processor and the storage medium mayreside in an Application Specific Integrated Circuit (ASIC). The ASICmay reside in a wireless modem. In the alternative, the processor andthe storage medium may reside as discrete components in the wirelessmodem.

In addition, the various illustrative logical blocks, components,modules, and circuits described in connection with the examplesdisclosed herein may be implemented or performed with a general purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general purpose processor maybe a microprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

The previous description of the disclosed examples is provided to enableany person of ordinary skill in the art to make or use the disclosedmethods and apparatus. Various modifications to these examples will bereadily apparent to those skilled in the art, and the principles definedherein may be applied to other examples and additional elements may beadded without departing from the spirit or scope of the disclosed methodand apparatus. The description of the aspects is intended to beillustrative, and not to limit the scope of the claims.

What is claimed is:
 1. A method of processing multimedia data,comprising: receiving digital interlaced video frames; and convertingthe digital interlaced video frames to digital progressive video framesby deinterlacing the digital interlaced video frames, wherein thedeinterlacing comprises: generating spatial information for the digitalinterlaced video frames and motion information for at least one of thedigital interlaced video frames; and generating the digital progressivevideo frames using the spatial information and the motion information,wherein pixel information corresponding to a first frame generated usingboth the spatial and motion information is weighted over pixelinformation corresponding to a second frame immediately succeeding thefirst frame and generated without the motion information.
 2. The methodof claim 1, wherein the deinterlacing further comprises: generatingbi-directional motion information for the digital interlaced videoframes; and generating the digital progressive video frames based on thedigital interlaced video frames using the bi-directional motioninformation.
 3. The method of claim 1, wherein the converting thedigital interlaced video frames comprises inverse telecining 3/2pulldown video frames.
 4. The method of claim 1, further comprisingresizing the digital progressive video frames.
 5. The method of claim 1,further comprising filtering the digital progressive video frames with adenoising filter.
 6. The method of claim 1, further comprising:generating metadata based on the converted digital progressive videoframes; determining an encoding parameter based on the metadata; andencoding the digital progressive video frames in accordance with theencoding parameter.
 7. An apparatus for processing multimedia data,comprising: a receiver configured to receive digital interlaced videoframes; and a deinterlacer configured to convert the digital interlacedvideo frames to digital progressive video frames by deinterlacing thedigital interlaced video frames, wherein the deinterlacing comprisesgenerating spatial information for the digital interlaced video framesand motion information for at least one of the digital interlaced videoframes, and generating the digital progressive video frames using thespatial information and the motion information, wherein pixelinformation corresponding to a first frame generated using both thespatial and motion information is weighted over pixel informationcorresponding to a second frame immediately succeeding the first frameand generated without the motion information.
 8. The apparatus of claim7, further comprising an encoder configured to receive the digitalprogressive video frames and encode the digital progressive video inaccordance with compression information generated by a partitionerconfigured to generate metadata associated with the digital progressivevideo frames.
 9. The apparatus of claim 7, further comprising adenoising filter for denoising the digital progressive video frames. 10.The apparatus of claim 7, wherein the deinterlacer comprises an inverseteleciner.
 11. The apparatus of claim 7, further comprising a resamplerconfigured to resize a progressive video frame of the digitalprogressive video frames.
 12. The apparatus of claim 7, wherein thedeinterlacer is further configured to: generate bi-directional motioninformation for the digital interlaced video frames; and generate thedigital progressive video frames based on the digital interlaced videoframes using the bi-directional motion information.
 13. The apparatus ofclaim 7, further comprising a partitioner configured to generatemetadata associated with the digital progressive video frames andprovide the digital progressive video frames and the metadata to anencoder for use in encoding the digital progressive video frames,wherein the metadata comprises compression information.
 14. An apparatusfor processing multimedia data, comprising: means for receiving digitalinterlaced video frames; and means for converting the digital interlacedvideo to digital progressive video frames by deinterlacing the digitalinterlaced video frames, wherein the deinterlacing comprises generatingspatial information for the digital interlaced video frames and motioninformation for at least one of the digital interlaced video frames, andgenerating the digital progressive video frames using the spatialinformation and the motion information, wherein pixel informationcorresponding to a first frame generated using both the spatial andmotion information is weighted over pixel information corresponding to asecond frame immediately succeeding the first frame and generatedwithout the motion information.
 15. The apparatus of claim 14, whereinthe means for converting comprises an inverse teleciner.
 16. Theapparatus of claim 14, further comprising means for resampling to resizea progressive frame.
 17. The apparatus of claim 14, further comprisingmeans for encoding the digital progressive video frames using providedmetadata associated with the digital progressive video frames.
 18. Theapparatus of claim 14, further comprising means for denoising thedigital progressive video frames.
 19. The apparatus of claim 14, whereinthe converting means is configured to: generate bi-directional motioninformation for the digital interlaced video frames; and generate thedigital progressive video frames based on the interlaced video framesusing the bi-directional motion information.
 20. The apparatus of claim14, further comprising: means for generating metadata associated withthe digital progressive video frames; and means for providing thedigital progressive video frames and at least a portion of the metadatato an encoder for use in encoding the digital progressive video frames,wherein an encoding parameter is determined based on the at least aportion of the metadata.
 21. A non-transitory machine readable mediumembodying instructions that, when executed by a processor, allow theprocessor to perform a method that processes multimedia data, the methodcomprising: receiving digital interlaced video frames; and convertingthe digital interlaced video frames to digital progressive video framesby deinterlacing the digital interlaced video frames, wherein thedeinterlacing comprises generating spatial information for the digitalinterlaced video frames and motion information for at least one of thedigital interlaced video frames, and generating the digital progressivevideo frames using the spatial information and the motion information,wherein pixel information corresponding to a first frame generated usingboth the spatial and motion information is weighted over pixelinformation corresponding to a second frame immediately succeeding thefirst frame and generated without the motion information.
 22. Thecomputer readable medium of claim 21, wherein the method furthercomprises: generating metadata associated with the digital progressivevideo frames; providing the digital progressive video frames and atleast a portion of the metadata to an encoder for use in encoding thedigital progressive video frames; determining an encoding parameterbased on the metadata; and encoding the digital progressive video framesin accordance with the encoding parameter.
 23. An apparatus, comprising:one or more processors; and a memory storing instructions, which whenexecuted by the one or more processors, is configured to cause the oneor more processors to: receive digital interlaced video frames; andconvert the digital interlaced video frames to digital progressive videoframes by deinterlacing the digital interlaced video frames, wherein thedeinterlacing comprises generating spatial information for the digitalinterlaced video frames and motion information for at least one of thedigital interlaced video frames, and generating the digital progressivevideo frames using the spatial information and the motion information,wherein pixel information corresponding to a first frame generated usingboth the spatial and motion information is weighted over pixelinformation corresponding to a second frame immediately succeeding thefirst frame and generated without the motion information.
 24. Theapparatus of claim 23, wherein the memory storing instructions whenexecuted by the one or more processors, is further configured to causethe one or more processors to: generate metadata associated with thedigital progressive video frames; and provide the digital progressivevideo frames and at least a portion of the metadata to an encoder foruse in encoding the digital progressive video frames, wherein thedigital progressive video is encoded based at least in part on themetadata.